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Following the closing of the National Library of 
Medicine's (NLM's) card catalog, a study was performed in 1982 to 
specify minimally acceptable requirements and capabilities for an 
NLM-based online catalog system and to evaluate the technical 
performance and user acceptance of available systems. Two prototype 
online catalog systems were selected for establishment, test, and 
evaluation: CITE (Current Information Transfer in English), which 
incorporates a user-friendly front end to the CATLINE system 
operating on the NLM's IBM 3033 main frame; and the public catalog 
access module of ILS (Integrated Library System), which also utilizes 
the CATLINE database but operates on a dedicated Data General S230 
minicomputer. Based on verification protocols, limits testing, a 
survey of library patrons — both users and nonusers of the online 
catalog — and two experiments involving searches by NLM patrons and 
Staff, it was recommended to the NLM director that CITE/CATLINE be 
adopted in its present state>»r use by patrons and non-^technical 
library staff, ajnd that appropriate systems support be provided. This 
final report briefly describes the study background, objectives, 
approach, methodology, an,d major findings. Extensive appendices 
comprise performance testing results, the results of analyses on user 
acceptance data, sample user and nonuser questionnaires, and a list 
of search questions and data collector's scoring sheets used in the 
search experiments. (ESR) 
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ONLINE CATALOG STUDY 
FINAL REPORT 

Elliot R. Siege! , Ph.D. 
National Library of "edicine 
Bethes.claj MD ,20209 



With tht closing of NLH's card catalog In tarly 1980. a coiButtF output «1ero- 
form catalog systtm (CONCAT) was tstabllshad on an Inttrin oasis In tht pilbllc 
catalog arta to provide patrons and non-ttchnlcal staff idth acoiss to iwir and 
updattd catalog tntrlts. (The Library's rtferencc and technical staff continued 
to use the CATLINE system for InforMtlon retrieval and file Mlntenanoe.) In 
December 1981. The Director assigned Dr. Elliot SI egel. Lister H1 11 Center, to 
overtake an objective and scientifically valid evaluation study of several 
alternative online catalog systems that had been proposed by In-house staff for 
adoption. These candidate |yst«ns iiere Intended for direct end-user catalog 
access, and could potentially fulfill the NLM's needs until such tlae as the 
planned MEDLARS enhancements are In place. The study was to be performed 
In-house utilizing a Study 6roup*/ad hoc working group aodel*. the duration 
was to be nine months. / 

II. Study lObjectlves ; 

1. Specify minimally acceptable requirements and capabilities 
for an NLN-based online catalog system; 

2. Determine whether we can establish, make accessible, and 
operationally test CATLINE (In Its present form). CITE. ILS ^ 

and Paperchase In the NLM public catalog areai ^ 

3. Evaluate the performance and user acceptance of each 
systffi. particularly among Library patrons and staff; 

4. Formulate a recommendation to the Director concerning 
the adoption of these systpis. Including specification 
of ariy modifications/enhancements required. 



*N«nbers of the Study Group were Nr. John Anderson. Dr. Clifford Bachrach. 
Hs. Lois Ann Colalannl. ¥ls. Karen Kameen. Dr. Henrqr Riecken. Dr. Uarren 
Selbert. Dr. Manfred Waserman and Dr. Elliot Siege! , Chalnwn. Important 
contributions were made by staff from throughout the Library: Ns.^larbara 
Bishop. Mr. Robert BorcHChoff. Ms. Pat Bosman. Mr. Pat Denman, Mr. Kenneth 
Grant. Mr. Gerard Guthrie, Ms. Doris Doran. Ms. Lou JCnfdit, Dr. Marjorit 
Kuenz, Dr. Craig LoCatls. Mr.^rnard SllversUln. Ms* Sally Sinn. fte. 
Barbara Sternlck. Ms. Beth Well . Dr. Michael Welsberg. Ms. Frelda Weise. 
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III. Approach: 



Upon MhlivMtnt of stu4y objtctlvts II ahd #2» two prototypt onlint catalog 
aystaas Mtrt salactad In March for tstabT1thMant» tost and avaluatlon. TIUQf 
art CITE ("Currant Infomatlbn Transfar In Englfsh"), Incorporating a uitr ' 
friandly front and to tha CATLINE systan oparating on tha Library's IIM 3033 
■ainfraiKi and the public catalog acctss aoduTa of ILS ("Intagratad Library 
Systan"), Mhlch also utillzas tha CATLINE database but oparatas on a dodlcatad 
Data Ganaral S230 nlnlconpiitar. loth CITE and ILS ara ongoing KID of forts 
In Sped all zed Infonutlon Services (SIS) and the Lister Mill Canter (LHC), 
respectively. The systwi designers. Dr. Tanas Doszkocs (CITE) and Nr. Charlts 
Goldstein (ILS), were requested to further develop and equip their cystans so 
as to conform to the technical specifications outlined by the Study Croup in 
objective #1. These would later constitute the performance criteria against 
which the systnns would be evaluated. In the case of CITE/CATLINE, this 
Involved continuing with the production and raflnanent of new software. For 
ILS, the principal hurdle was to create a functionally acceptable daUbase 
suitable for operational use by Library patrons during the test period. This 
Involved the conversion of nearly one-quarter nITIlon CATLINE records to NARC 
format, and their loading and Indexing on ILS. The tine allotted for these 
activities was very short given the planned duration of the Study* CITE/CATLINE 
was successfully established and made ready for test and evaluation In April, 
and ILS In June. Concurrently, other working groups were proceeding with 
development of the study design and data collection Instrunents. 

IV. Overview, of Study Design ; 

The study design provided for the Independent and comparative assesspents 
of CITE/CATLINE and tha public catalog access nodule of ILS on two dlnen- 
slons: technical performance and user Kceptance (I.e., effectlMness 
from the users' standpoint). The study Is unique, nethodologlcaily. In 
that It evaluates two online catalog systems within the same operational 
environment, utilizing the same staff, computer terminals, databases and 
patron population. 

Technical perfonnance was addressed by means of specially constructed 
verification protocols which systematically determined and documented the 
availability and performance of all required system features and attributes 
specified by the Study Group In a formal "requirements" document. Of 
particular Interest were the search and display elements of CITE and ILS, 
and certain aspects of "user fr1end^1ness^ (Both systems have been 
designed for direct and-user access.) This assassment of technlcaV 
perfonnance also made use of "stress* or "Units" testing which sought to 
elicit additional dati on the outer ranges of system capabilities. All 
testing was carried out by a two person team highly skilled In online 
systnn use, and previously unlnvoived In the design and development of ^ 
CITE and ILS. 

The concept of "user acceptance" Is nore difficult to operationally define 
and measure. In this study ^ user acceptance was addressed In three ways - 
tach complementary In scope and methodology: 
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1. W«1hUtrit1on of tht 60-1ti« ohjtctlvt ^oni»t/CW- sunra^ qut^ 

to pitroos liho conducttd • catalog starch on tltMr cut or ILS <ii-30Q m 
taJTrntom). For this puipott/ CITE ws Mdt avail aMt for pat w teetft 
SMii^latt- April and May; and US (without CITE) was aval labia thru »ft 
of tht suimitr. Tht ustr dutstlonnalrt characttrlits - anong a larft 
nui*tr of bthavloral and attltudlnal varlablts - t»»«,"«t"f*^«L!f'*'L*i. 
starch rtqulrtntnts, salltnt dtmographlc factors, and »5J»^«2^"*.?'J5 
sptclfic tnttrfact ftaturts and starch outcoMS. Co«p|tt1oi» t1«t If 15 
nihutts; wt txptrltnctd a compllanct rata bttttr than m, A ^•S'" 
■non-ustr" qutstlonnairt was also adulnlsttrtd to a sanplt^of «-3W^ 
patrons who had not ustd tithtr onTlht catalog. Thtst qutstlonnalrts 
wtrt dtvtloptd as part of tht Council on Library Rtsourcts (aR) 
natlonwidt study of 17 onlint catalog systms. Ust of tht ««2 
Instruments, with saiM nodi fi cations to iNtt sptclfic HUi nttds, ptnits 
coRvari sons with tht findings of that study. , 

2. Conduct of a "Sample Search Emrmtnt" which systematical ly txpostd 
ipantl of 20 NLM staffers (Hbrarians and non-librarians) to both^ . 
CITE and ILS under control ltd fitld ttst conditions. Tht txptrimtntal 
stssions simulattd rtprtstntativt usts of an onlint catalog across 
six cormon starch types (personal and corporate author, conftrtnct, 
stritsVtitle, and sSject). He utilized 14 sptcially^st tcttd 
UlrS'qutry pairs - mitchid by type and level of diff culty - jo 
that a respondent did not repeat the same search question on both 
systems.- Dependent measures of particular inttrtst art ustr ptr- 
ctptions concerning the quantity and rtltvanct of I"^;"**!*!.^,-, 
rttritvtd, ease of use, adequacy of screen displays, and prtftrtncts 
(If any) bttween systems by search type, and overall. User fttd- 
back on specific system 11 kts and dislikes, along with suggestions 
for features not now present, were also obtained via "ructurtd , 
post-search interviews which, all together, "v^WdJ 1/J hJfrs^JJJ 
session. The -Sample Starch Exptrimtnt- was conducttd with NLH staff 
ustrs during the month of August. 

3. In September, we were in a position to ««^jb2hCITE ;«d ILS simul- 
taneously available to NLM patrons in an effort to disctm thtir 
liJS" dtsllkts aid systtm Jrtftrtncts afttr having had an mon^M^f 
to coiduct a starch of thtir own choosing •".ft*'* 
systtms. What we sacrificed in exptrimtntal control wt Joptfully 
MdeTp in authenticity. Sixty patrons participated 1" this £ojt 
oarison S earch Experiment ." which utilized a number of the »«^ . 
K?uctur3 interview quest ions used in tht rf^'tivtlyjre control ltd 
but Itss rtalistic -Samplt Starch Exptrimenf conducted with »taff, 
S bits tStriments, order of prtstntation of iyji^L^Srl dJSISJ 
for -odd-tvtn" ustrs, and printouts of starch activity wtrt productd 
and rttaintd for subsequent analysis. The latter tnabltd us to 
SlatriSual ustr starching bthivior to stattd systtm prtftrtncts 
and starch outcomts. 




V. SuMMfy of Mi Jot rindinot! 
Ustrs of the Coreputtr Citiloo 

o Uitrs of the computer catalog represent a broad cross section of 
professional roles/occupations 

® fS*!*"**"* computer catalog are Infrequent or first time 

visitors to the Library 

Characteristics of Catalog Searches 

0 Most users of the computer catalog come idth subject-related 
Information 

D Most users of the computer catalog are looking for books on a topic 
0 Most users of the computer catalog search by subject or topic 
uter Catalog vs the Card Cataloo and COHCAT 

o The computer catalog Is preferred to the card catalog and COMCAT 

0 CITE users prefer the computer catalog to the. card catalog In 
higher numbers 

0 Among patrons who have used CONCAT, preference for the computer 
catalog Is equivalent for CITE and its user* 

User Satisfaction with CITE and ILS 

0 More Information Is found by users of CITE 

0 Satisfaction with search results Is higher among users of CITE 

D Overall satisfaction with the computer catalog Is higher among 
, users of CITE J ' 

o Professional role/occupation is unrelated to satlsfatlon with CITE 
end lis 

Preference for CITE/ILS and Search Typ e 

0 CITE is preferred for subject searching , 
o CITE Is preferred for title searching 

o No clear-cut preference for CITE/ILS In perfonalng other types of 
known itm searches 
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CITE/ILS Pltplm 

o CITE and ILS displiors art. ovtrall. aqually acctptabia to uttrs 

Cm/IL5 Instructions 

0 CITE Instructions, prompts, "halp" wssagas ara sowwhat mn 
tffactive 

Tachnical Parforwince of CITE and ILS 

Parforwnca mting rasuUs, saparately corroboratad by findings fro* tha 
thraa usar studi as. indicata that: 

0 CITE has no critical shortcomings and is assantially acoaptabia 
as is to the large mijority of catalog users ' 

0 ILS meets most required system featuras/attributas, and performs 
more than satisfactorily on most txpas of catalog searches. 
However, its performance on subject-related searches can be 
substantially improved by the inclusion of several capabilities 
most value has been demonstrated (e.g.. greater correspondence 
between descriptor and text word searching, ability to search on 
multiple terms simultaneously, automatic weighting of retrieved 
records by frequency of search terms occurrence).* ' 

VI. Reconinendation : 

2?*!?J!5??L^?' it was recommended to the Director.- and accepted, that 
CITE/CATLINE be adopted asjs for use by the NLM's patrons and non-tachnical 
staff, and that appropriate systems support be provided. 

In addition, a very brief list of desirable system enhancements was presented 
to Dr. Doszkocs. along with a request for resource estimates to be prepared by 
him for review and approval by ^LH management. 

AVecommendation for the adoption of CITE/CATLINE is consistent with several 
related operational considerations: accept ablr reliability and stability of 
present CITE/CATLINE software; likelihood that NLN will pursue development 



*Nr. fiotdstain will be presented with complete Study documentation concemtng 
the tacnnical performance and user acceptance of the ILS public catalog 
access nodule, including specific suggestions for enhancing the user Inters 
face. This will serve to make the ILS. of which the public catalog access 
module Is cnly a single part, an even more attractive product for the nation's 
library community. 
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of "CITEHILL" Indtptwltnt of thii study's outcoM; additional rtsourcts ftqufrid 
to support optratlon of ILS on dtdlcattd ninlconputtr; and additional rtsoyrcos 
rtqulrod to support nalnttnanct and updating of stparate ILS databast. 

teptndlxtS ! (stparattly avallabU) 

A. Ptrformance Tasting - Prtstntatlon of Findings ^ 

B. Ustr Acceptance - Results of Data Analyses Across Methods 

C. CLR User Questionnaire (NLM Version) 

D. CLR Non-user Questionnaire (NLN Version) 

E. Sample Search Experiment - Search Questions (Odd/Even Versions) 

F. Sample Search Experiment - Data Collector's Scoring Sheet 

6. Comparison Search Experiment - Data Collector's Scoring Sheet 



Elliot R. Slegel. Ph.D. 
December 1982 
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APPENDIX A. 
Perfonnance Testing - Presentation of Findings 
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Verification Testinq - Presentation of Findings 

» ■ ' * 

The attached materiaV contains the foil owinq sections: 

A) Background and Methodolopy 

B) Search/Access Features 

-Sunmary chart outTining the "R" and "N" status elements 
for which no search/access is orovided or for which access 
is limited 

-Sunmary of the "R" and "N" search/access features 

-Verification testing data - Discussion of search 
characteristics by element for each prototype system , 

-User cordiality 

C) Limits Testing * 

D) Display Features 
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Task 2. 

'objective 1. 

Objective 2. 

Objective 3. 



Assess Technical Performance 

of Prototype Systems (CITE and ILS) * ^ 

Verify presence/availability of required ("R) prototype system 
features and certain necessary ("N") features converted f rem "R* 
status 

Document performance characteristics noted in verifying the 
features/capabilities. • 

Assess system performance capabilities with selected searches 
representing ambiguous or complex queries. 



Background 

The document "Requirements and Capabilities of an NLM^Based Online Catalog 
System" developed by the Online Catalog Study Grojup, February 24, 1982 formed 
th© basis of the set of requirements to be verified^r The specifications 
outlined in this document were drawn from several sources*, including the 
fjrior work of the NLM/LO Task Force to Evaluate Online Capability for Public, 
Use, January 1982. The specif icartioris addressed the following categories of 
system attributes: 

Data Base Contents 
. • Composition of Records { 

Search/Access Features . ' » 

User Cordiality ' .\ 
" , Display Features 

Auditing and Reporting. 

Verification testing procedures were developed for the features'of search/access, 
user cordiality, and display. No. verification testing procedures were developed 
for the attribute "Composition of Records" sipce the full set of required 
elements was reflected in the search/access features and the display features. 

J'' ' ' 

The original requirements, categorized as R - Required for test, N- Necessary 
for operational system^.-and 0 - Optional were developed with the input of Library 
staff in Reference Services Division and Technical Servf^es J)i vision. At the 
same time, the Study Group was considering the issues of which available online 
systems to include in the evaluation. The four candidate systems were: ELHILL 
CATLINE, CITE, ILS, and Paperchase (Beth Israel HosiJital) » - 

In the course of soliciting resource estimates for preparing the candidate 
systems for testing, CATLINE .and paperchase were ruled out as candidate 
systems on the basis of the resources required to mike Paperchase availaWe 
at NLM with the full set of CATUNE t'ecords and ^he technical resources 
required to effect any significant xhanges in CATLINE (ELHILL) at this time. 
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The requirement's document was sent to the system developers of CITE and JLS», . 
Dr. Tamas Doszkoes and Mr. Charles Goldstein.- respectively. They were'^asfced 
to prepare resource estimates for providing prototype, t«st versions of their - 
systems and. estimates of resources required to incorporate the necessary , • . 
features for an operational system. ' . 

Direct negotiations were undertakpn by the Study Group with, Dr. Doszkocs_ and 
Mr. Goldstein to work out an acceptable set of requirements for each system 
within the constraints of the study timetable and avaiUble personnel. This 
involved the shifting .of a few features from "R".to "N" status. In £t her 
instances, the system developers were requested not to proceed with the 
implementation of certain features because of ,the time and resourqes required. 
In the judgment of the Study Group, the testing of both systems Would still 
be made against a comparable set of performance requirements. For each system, 
a written "contract" was prjpared, clarifying which features could be ^ 
accommodated within reason in a test version and which features are feasible 
for ftffyre development of a potential operational system. 

The revised list of features far CITE and ILS was used Jiojaotf-tfy or annotate 
the requirements list in preparation for verification testing. This revised 
document, ca,lled the "Features List" was reviewed by the Online Catalog . 
Study Group and became the" basts of^the checklist used ipf verification 
testing. The verification tes/ing was designed to determine objectively, 
whether a given feature is present in either prototype system.. Certain 
performance characteristics of each systenv^t ii, haw well or to what, 
degree each system performs-or satisfies the feature or.>fetr^te,. are ; 
also to be noted in completing the tes\ searches. However, to fully 
evaluate "maximumi^performance, a t+iird *^€t of test parameters, called 
"limits testing", was developed. .Limits testing is designed, tp elicit 
additional comparative data an each system in the handling of conjplex 
or ambiguous queries and queries known to retrieve large sets (.numerous 
records). ^ . 



Methodology: 



J 

Verification testing is the objectiye determination of whether a feature iS 
present or not 'present in either system, CITEtTr ILS. A methodology was 
developed by the Working Group on Performance Testing (Ms. Sinn, Ms. Kamean, 
Dr. Siege!) for verification testing, of each of^ three sat^ones of 
attributes: 1) Searcli/access features; 2) User cofdiality; and 3) Display 
features. The limits testing task was incorporated as a. Second phase pf 
the Search/access verification. ° 
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1) Search Access 
a) Objectives: 



Verify presence/availability of "required" and "necessary 
search elements. Verify the'' avail abil ity of index displays, 
textword search capabilities', truncation, to Iterance for / 
variations in spacing, punctuation, spelling arid special 
characters. 




b) Methodology: Elements were sele{:ted from fxistTng source CATLINE 
' ' records known to |)e present in both systems, h total 

of 11 CAJLINE reQ<irds containing at least one of the 
4' searchable elements being verified were selected to 
' include both monographs and serials, English knd 
foreign languages and a range of entry dates ; 
(1976-1981) not to exceed the CATLINE subset already 
loaded for the ILS prototype system (as of 3/82 this 
included 1976-198T). 

A full, detailed printout of each designated CATLINE 
record was obtained and annotated to indicate which 
elements were to be searched to retrieve that specific 
record in each system. In severa;i cases, more than 
orie record was used to test for afi individual element, 
far example, each record contains a title, so several 
represeptative types of titles were employed in the 
search/access verification. The same elements from^ 
the same set of CATLINE records were checked in each ' 
system. 



A checklist document was developed to 

CATLINE record printouts. The check! 

space for d&cumenting the following t 

about each search element: 
Directly searchable 
Specific record retrieved 
Number of steps required - 



Index displayed 

Element is textword searchable 
Number of steps required 

Element can be truncated 

Number of steps required 



accompany the 
ist provides 
nformation 

(Yes/No) . 
(Yes/No) 

(Specify how many 

user responses/ 

actions necessary 

to retrieve specified 

record) 

(Yes/No) 

(Yes/No). 

(Specify) 

(Yes/No) 

(Specify) 



For each element searched or feature tested, the 
verifiers were also asked to provide comments, a^ 
appropriate* noting any system characteristics or 
problems that either significantly enhanced or 
hindered the search/cetrieval . Members of the 
Working Group reviewed verification testing 
results while testing was in progress in order 
to resolve any noticeable discrepancies in the 
findings and to clarify procedures and approach. 
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c) Verifiers: Two verification testers were selected from Library 
Operations' staff. A third individual was designated 
as alternate or back-up in case of scheduling problems 
with the primary verifiers. These individuals 
(Lou Knecht, MEDLARS Management Section, Pat Bosma, 
Selection/Acquisitions Section, and Barbara Bishop, 
* Cataloging Section) are each experienced and competent 
in the use of online bibliographic retrieval systems 
and are familiar with cataloging and bibliographic ' 
information as it is reflected in CATLINE, but none 
had prior involvement in the development of either 
CITE or ILS nor any familiarity vin using either of 
the prototype online catalog systems. 

,. ■ . , ■ ' V 

2) User Cordiality/Prompts/Online-HELP features- 

Objectives: Verify presence/absenj:e of system prompts, menus, online 
HELP features 



Methodology: Obtain some information on the availability of prompts, 
menus and index displays fri>m the Search/access 
verification testing. On a separate checklist, verifiers 
were asked to note all junctures at whicti'the prototype 
system automatically generates prompts or HELP messages and 
those which must be user-generated. Verification of user 
cordiality also included comments on the appropriateness » 
and clarity of all HELP messages. 

Verifiers: Same verifiers as for the SearchMei^ss (Ms. Knecht, Ms. Bosma), 
with additional comments from S^inn, Working Group on 
Performance Testing. / 



3) Display Features < \ 

Objectives: Verify the- presence/absence of "required" and "necessary" ^ 
elements in the brief and full record displays 



Methodology: The presence of required and necessary elements in the 
brief and full displays was verified by obtaining print- 
Outs of the source CATLINE records used for verification' 
of Search/access features augmented by additional sample 
records selected from CATLINE. It should be noted that for . 
the ILS, the standard print display was defined as "bfief" . 
and the MARC- formatted version was defined as "full". 



Verifiers: S. Sinn, Working Group on Performance Testing 

O ■ ■ ■ ■ ■ A-4 ■ ■ ■ 
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4) Limits Testing 

Objective: As$ess system performance ce^abilities with selected 
searches representing ambiguous or complex queries. 



Methodology: Representative author, title, series, and subject searches 
were constructed based, in part, on typical kinds of 
problems encountered in searching other online systems. 
Other factors considered in designing the limits testing 
were the contents of the GATLINE data base and the 
existing capabilities/deficiencies of ELHILL CATLINE." 



Verifiers: Same verifiers 'a.s for Search/access and User cordiality 
(Ms. Knecht and Ms. Bosma) 
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SEARCH/ACCESS 



This, chart briefly sumnarizes the "R" and "M" elements for 
for search or for which the search capability is limited. 



CITE 

(Required "R" elements) 

1. Subheadinqs (SH): Subheadings.-topica1 , neoqraphic, 
language, ^nd fomijr-are not searchable ' 



2.1n1t1ali^ear (Yl): Element is not directly 
searchable, but can be used to limit the number of 
items. for display after retrieval of the initial 
s^. ilser can enter the earliest publication date 
of interest. 

3. Final Year (Y2): Element is: not searchable or 
available for narrowinn search 



4. Unique Identifier (UI): Element is directly 
searchable, but this capability is not apparent 
to the user. Specific record can be retrieved 
by the Unique Identifier if it is entered as a 
search tern under "Series, Conference Name, 
Organization, Call Nunlber" 

erJc 
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search/access that were not available 



■ M ■ . ■ 

(Required "R" elements) 

1. Subheatlinqs (SH): None of the subheading 
types, topical, geographic, lanauage or form, 
is directly searchable. They are displayed 
in the index of subject headings and may be 
selected as a subject heading qualifier to.' 
limit retrieval to a more specific topic* 

The subheadings used for a given subject head- 
i no are dispTayedsin two alphabetical lists, 
topical subheadings fiVst, followed by the 
aeon.» lang. and form subheadings as a group. 

2. Initial Year (Yl); Element is not directty 
searchable, but it may be used to narrow 
search if the original retrieval exceedis W 
hits. The technique for narrowing a search 
by publ ication date is not well eJ^Plaifjed i" 
the HELP message for "Narrowing Search (/NS) 

3. Final Year (Y2): Element is not directly 
searchable, but mav be used in combination 
with Initial Year (Yl) to narrow a search. 
The actual procedure for limiting searches by 
publication date is not well ecplained in the 
HELP messages. 

4 Title (TI): Only the first occurrihceot the 
title field is indexed and searchable. .Variant 
titles and titles of related works given as 
other occurrences of the title field (TI In 
CATLINE, 740 in MARC) are not retrievable. 

5. Call Number (CA):- Call' Number of the. Dashe<J-On 
Entrv field is not searchable. The Da shed-On 
Entry field is not converted for M.ARC . -jj^ 



CITE 



(Necessary "N" elements) 

' ■ . • . ♦ ■ 

1. ISBN (BN): Specific ISBN is directly searchable 
if entered exactly as stored, but this search 
capability is not apparent to the user. ISBN 
inust be entered under se&rcb category for Series, 
Conference Name, Organization, Caul Number. 

the system displays the ISBN as searched with- 
out any ^element qualifier. 

^ ■ ' ■ fl 

2. Country (CY): Not searchable 

3. ISSN (IS^: Directly searchable if entered exactly 
as stored, but search capability is not apparent to 
the user. I,ike the ISBN, the ISSN is searchable 
only under category "Series, Conference Name, 
Organization, Call Number". The system displays 
the ISSN as searched without any element qualifier. 

4. Item Type (IT): Not searchable „ 

5. Language (LA): Not directly searchable, but the 
val lie English is available for limiting the 
number of records to be displayed after retrieval. 
User can choose to dispaly English language items 
only. 

• ■ • 

6. Media (ME): Not searctwble 

7. MeSH Tree Number (HN): Not searchable 
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10/20/82 



as 



(Necessary "N" elements) 



1. ISBN (BN): Element is searchable, but it must ; 
be input without the hyphens separating parts of 
the number. The format for storing ISBNs in 
flARC is without hyphens and the field was converted 
this v/ay for ILS. This format characteristic ^is 
not well expla,ined in the ILS system, and the problen 
is compounded by the inconsistency. in ISSN search , 
which must include the hyphen. 

2; Country (CY)^ Not searchable 

3. ISSN (IS): Seiarchable, but not consistent with, 
the ISBN field 



4. Item Type (IT)' Not searchable 

5. Language (LA.): Not searchable • 



6. Media (ME): Mot searchable 

7. MeSH Tree Number (MN): Not searchable. This data 
i s not converted for MARC . 
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FBATURE/ ATTRIBUTE 



STATUS 



CITE 



Boolean logic on Indexed 
fields ("and" "or") 



Can logically combine 
elements in initial search 
statement 



Can browse indexes by: 
Call Number (CA) 



Corporate Name (CN) 

Corporate Naijie as Subject 
(CS) 



MeSH headings/Subheadings 
(MH/SH) 



R 



No true Boolean 
capabi 1 1 ty 



No 



Yes 



Yes 

Yes- Displayed in same 
index as corporate^ 
name (CN) 

No 
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COMHEMT 



No true Boolean 
capability. Can 
combine certain sets 
in a secondary 
"narrowing" search, 
but this does not work 
for all search categorji 

Only author & title as 
author/title search 
key 



Yes 



Yes 

Yes - Separate index 
from CN. Search as 
Subject. 

Yes, subheadings are no 
indexed seaprately, but 



ies ■ ' 

In both CITE and ILS 
can only combine term; 
from the same element, 
e.g. words in title 



inaexea seaprateiy, uut f,.^ ^_ thp TLS index 
as qualifiers applied t( J ^" J^-^ 



specific MeSH heading, 
two levels of index 
display: MeSH headings, 
and, upon selection of 
a MeSH heading, will ge : 
Index display of the sul 



CITE: Index display 
results If search 
tenn(s) are right 
truncated sofflclentlj 
to make them nonunlqw 
i.e. the equivalent 0 
a multt-meaning messace 
In ELHIU 

ILS: Initial search 
queries, even If 
unique and only one 
"hit" always result 
In index display 
There Is a major deslch 



displays. The Index 

"window" begins too far 

forward in the alphabet 

when no exact Match Is 

f oupjd. .User cannot, bj )w 
lex frow-tnat 
point. 



. found. .Use 
.•up ihe imt 
l^htfadTngs. 



21 



fEyniRE/ATtltlBOTE 

(Index browse « Con t.) 
Personal Name (PN) 
Personal Name as Subject (PS) 



Textword Searching : 

Corporate Name Title Series 
(CE) 

Corporate Name (CN) 

Conference Name (CNA) 

Corporate Name as Subject 
(CS) 

Personal Name Title Series 



Series Title (SE) 



Title (TI) 

Title as Subject (TL) 

Uniform Title as Added Entry 
(UQ) 

Uniform Title (UA) 
Uniform Title as Subject (UTSb 



R 
R 



R 

R 
R 



R 
R 



ERIC 
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Yes 

Yes- Same index as 
Personal Name (PN) 



Yes 

Yes 
Yes 
Yes 



(Unable to verify, 
but should be treated 
same as SE and CE) 

Yes 



Yes 

(Unable to verify) 

(Unable to verify) 

(Unable to verify) 
(Unable to verify) 



Yes 

Yes - Separate index 
from PN (Author searcJ 
Search as Subject. 



Yes - Conditional 



Yes - Conditional 



Yes - Conditional 



Yes - Conditional 



(Unable tb^ verify, 
but should be treated 
same as SE and CE) 

Yes - Conditional 



Yes - Conditional 

(Unable to verify) 

(Unable to verify) 

(Unable to verify) 
(Unable to verify) 



) 



CITE: textword search 
works only if entered 
under category of 
subject /search (Subjei 
search > in' CITE uses 
subject headings and 
textwords. The text 
words come from the 
CATLINE ELHILL TW indix 
which includes: TI,TC, 

as,se,ce,pe.cn,cs,cna: 



ILS: textword search 
can only be done with 
single term entered a 
term search. Cannot 
Nperf orm textword sear4h 
^nder other categories 
such as author, title, 
unless the textword 1; 
the entry point of th« 
heading In the index. 



FEATURE/ATTRIBUTE 



STATUS 



CITE 



TLS 



5. 



Can display lists of 
related search terms 



, 6. 



Call enter search term 
"right truncated" 



ERLC 
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No - CITE will display 
ranked terms in 
subject search based 
on their frequency in 
records., these may or 
may nbt be related. * 



Yes - CITE "stemming" Yes 
routines in subject 
search automatically 
perform some right 
truncation in the coUr^e 
of selecting additional 
iearch terms to prdces;. 



No - all "lists" are 
alphabetical index 
displays , . 



♦Verifiers commented on 
their preference for 
selecting terms to be 
included rather than 
excluded. 



COMMENT 



B-5 



Authority file links 
for subjects and 
name/series, includi(ig 
see-re liated^and see 
also references were 
not required for 
prototype systems* 



In both systems, 
truncation requires 
no special symbol 
(as in the use of 

th6 colon {%) in 

ELHILL) 

Cannot truncate term ; 
embe(Jded in the 
searGh phrase 
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mTIWE/AmiBUTE 



STATUS 



User Cordiality 

So^are provides: 
prompt/menu mode 



coMnand mode 



R 
N 



swHchable menu and command N 
mode 



Online HELP facilities: 
System initiated 

User initiated 



R 
R 



0 

ERIC ' 
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CTTF. 



TL!? 



COMMENT 



B-6 



Yes - Mixture of 
prompts and menus 

No 



No 



No 



Yes - Same set of HELP 
messages available 
at any juncturie or upoii 
command at any time 



Yes - Mixture of prompts 
and menus 

Yes - For some searches 
and functions. The 
"Ref" display will 
differ somewhat if 
pers . name i s searchec 
as Author from menu 
choice rather than as 
command 7AU= " 

Yes - For some types 
of searches there is 
an option to select 
from menu or enter 
a command 



No 



Yes - CITE HELP messaeids 
are geared to coincide 
with the type of actic 
requi red or search 
performed. Appropriate 
to the type of search 
and actions required 
of user 



ILS: Menu has no 
option for NONE, user, 
must recognize that 
START or STEPBAGK 
should be used to 
exItJ from menu or 
indek display 

ILS: Provides 2 differ- 
ent syntaxes for inpu 
in searching pers. 
names. As Author seanh, 
the parts of name are 
entered on formatted 
screen, without comma! 
as punctuation. As 
subject search or in 
command mode {/AU=), 
the name must be inpu 
with correct punctuat on 
and spacing. The 'REI ' 
display is different 
in these cases. 



Both systems do 
generate some 
suggestions or messages 
as guides to the user. 
These are not separate 
HELP messages but are , 
part of the prompt/ \j 
message programs. 



ILS: There are "loops" 
in the sequence of 
HELP messages tlhere the 
user can get caught 
tvA the only way to 



rEATliRE/ATTRlBUTE 



System ,1s lenient with 
regard to Inconsistencies 
In syntax 

Spacing 



Punctuation 



Order of words 



Comp1etenes$ of name 



Variant spelling 



6-7 



GTTR 



TLS 



COMMENT 



No 



No 



Yes - For subject 
searches 



Yes 



Yes - to a degree, only 
under Subject search. 
Tenns hot found in the 
1 ndex are di spl ayed to 
user for response-- 
they may be re- typed 
in case of typo or 
ml sspel 1 i ng , or omi tte( 
from*the search at the 
user^s discretion 




CITE: Win tolerate 
some Inconsistencies 
In spacing and ^; 
punctuation only. In 
subject search. It Is 
rigorous In requiring 
exact Input for serie: t 
names I call numbers, ftc. 

ILS: Intolerant of 
Inconsistencies in 
spaci ng ajfid punctuat1([n 
for all searches exceift 
Term (which must be 
single word anyway) 



|See truncation) 

Neither system will 
jtolerate Incomplete 
terms Imbedded In a 
name or series search 
e.g.. Natl Lib of Med 



FEATURE/ATTRIBUTE 



STATUS 



CTTR 



COHMENT 



Spetlfic Online HELP 
messages 

For Log-On 



For printing hardcopy 



For initiating search 



For author search 



(These specific 
HELP messages 
were tested 
but were not 
specified as 
sujph in the 
requirements 
document) 
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Good - clear and 
concise - 



None online 



More information in 
the HELP messages to 
suggest how system 
processes queries coul 
help user formulate 
strategies. The HELP 
i nstructs user on wha 
to do next or how to 
respond to system, bu 
does not explain any- 
thing about the logic 
of various search opti 

Sufficient 



Clear, but should be 
explicit about use of 
the carriage return. to 
start. "Press CR to 
continue" is not clear 
when user hasn't even 
begun. HELP feature 
is described different 
on 'green' sheet vs. 
online. 

None ortline 



Online HELP message 
says enter /, 7ES, or 
/NS, but CMD prompt 
only lists /ES and /NS 



)ns 



Onl ine HELP message 
does not adequately 
instruct in use of 
coninand mode, e.g. /AU 
as shortcut to the 
selection from menu 
HELP message should 
cl ari fy that ^nly one 



REP I may bv 
a time. 



btilnettiU 
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mtURE/ ATTRIBUTE 



STATUS 



CTTR 



ILS 



COMMEtrf 



(HELP inessa<ies Cont. ) 



For title search 



For subject search 



For textword/term s earth 



Other 



To narrow search 



Suggest alternate search 
strategies . 



Sufficient 



HELP messane could 
provide more auidance 
in what is included in 
subject searchinq 



None except as part 
of subject search 
Source of textwords! 
(titles, niames, etc.) 
should be explained 
to aid the user in 
formulatino best strattny 



None 



No e)tplicit HELP 
njessage— see prompts 

Nothinn explicit 



Sufficient 



Difference between 
subject search (MESH) 
and term search should 
be explained 



Messaqe should be 
cl earer in descri bi nff 
term as single word 



CITE HELP messages 
were found to be too 
bri ef and presume sm i 
kniK^ledge of catalogii 
for effective searchii 
of series* call. #, or#an. 
name, etc. (type 3) 



Ca 1 1 Number - requ i red 
two steps to get to 
HELP message explai nine 
call number 



Narrow search options 
clearly explained 

None 



in 
ig 



Verifiers requested 
more instructions on 
how to exit a search 
i.e. , abort, and how 
to logoff or end a 
session 



Both systems could be 
improved by better 
explanation of subjeci 
versus term/textword 
search. Use of terms 
or textwords as alterrate 
methods when other searches 
fail. 



Limits Testinn 



Throuqh perserverance and creative searchino by the testers, there were 
no real system "failures" detected for either ILS or CITE. Some response, 
if not always relevant retrieval, was always obtained and neither system 
"collapsed" under testing. There were differences in performance 
characteristics and overall time required to complete some types of searches 
between the tyo systems. ILS frequently required more steps (user actions), 
to complete an ambiauous search, whereas the use of subject search on CITE 
allowed natural language phrases as input in one search formulation. 

Both svs tens did exhibit some shortcomings" or deficiencies in various 
categories of limits testing. These shortcomings are not necessarily^ 
related to "stress" factors, such as the size of retrieval or combination 
of highly posted terms. Many of the same shortcominns were also detected 
in the search/access performance testina. 

The results of limits testinq on name and subject search! no ^clearly point 
out the desirabilitv of mappinp to authority files for variant forms of 
names/subjects (See De la Cruz example under personal author category 
and Alzheimer's Disease under subjects). 

Difficulties in achieving desired retrieval corporate authors and^ 
series in both systems can be attributed to the lack of tolerance for^ 
inconsistencies in spacing and/or punctuation uDon input. This system 
riqidity can be partially overcome by utilizing the term search in ILS ana 
subject searbh in CITE as the testers were cdfesionally required to do 
in order to retrieve any records. 

The systems are nearly equivalent in their ability to handle^personal and 
coroorate authors with conrion (nonunique) names. Both provide the user 
with an index display, but CITE will permit the user to^select all entries 
of the name from the index while ILS requires the user to specify each one 
individually. , 

CITE failed to display as many title natches under "Who's who" fs ILS. 
This may be a result of the stbpword list used in CITE or a problem with 
the stemning routines. 

In subject searching, the capability of^comblning concepts J^n tje^initial 
search query oh CITE made subject searching of multiple terns easier than the 
S S&ining subjects/tems on ILS. ILS achieved greater Precision 
in subject searching with cataloging subheadings than CITE since these 
subheadings are indexed with the MeSH headings in ILS. 

CIT^has the capability of providino the user with some variant spellings 
hmong the ranked tem display in the subject search. However, this same 
capability does not exist for other search categories on CITE. 

The /word order of subject term's is not significant in the subject search 
ontlTE Because the subject search in ILS is by MeSH terms Ion y. the 
word order is critical and must natch the word order of the headinn In the 
Index. 

Neither system tolerates variation in word order und^r name and series . 
searching. j.^^ • 



LIMITS TESTING 

nR«;FRVATTnns/RES»JLTS 



Category' 



Personal Name 
Common Surname 



Connon Surname 
with 1st initial 



Compound Surname 
(Surnames composed 
of multiple "wordsf 
or prefixes such 
as "von", with 
or without hyphens) 



Diacritics 



o 

ERIC 
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ILS 



No initial count on number of matches in index 
Begins index displav of personal name headlnns 
from top of index alnhabetically at a rate of 10 
at a time, forward in index (backward only to 
point of entry in index) 

Gives number of cataloq records for each unique 
headinp, but not the sum total 



Enterina forename initial will narrow the search 
to entrv at the point of the index of surnames 
with that initial. In ILS, this reduced the 
number of index entries from 33 to 31 

In order to know totaf" number of records retrieved 
user must browse index iand count the number of 
catalon records for each unique headinn 

To narrow search with forename tnittal or part 
of forename, user must benin a new search 

User is always given the index display even if 
name headinq entered is unique in the index 



Surname must be entered exactly as stored in the 
index, with correct word order and punctuation 
and hyphens 



Not stripped out of data. Improper translation 
of diacritic characters in the ILS loadinn makes 
retrieval on terms with diacritics Impossible^ 
unless user knows what characters the diacritics 
were mapped to 



CITE 



Gives initial count on number of matches in indej 
Beains index display of personal name headings 
from top of index alphabetically at a rate of 
5 at a time, forward and backward limited pnTy 
bv reachina end' of index. ^ . 

GiVes number of catalog records for each unique 
heading, but not the sum total 



Entering forename initial narrowed search 
from 828 ind^x terms to 65 

In order to know total number of records retreivc 
user must browse through index and count the 

'nuitiber of catalog records for each unique 
headina 

- - ■ . ^ . " ■ ■ • 

To narrow-search with forename initial or' part 

of forename, user must begin a new search 
User is given index display only when name 

heading is not unique in the index 

Surname must be ehterea~exa(CtTylirstor^^ 
index, with correct word order and punctuation 
and hyphens. A comma must be ihserted at^Jhe 
end of the surname, otherwise, system wtlJlnse 
the cdnmaas a default; for compound surnames 
the svstem presumes that the first "word" (up t 
the first soace) is a simple surname and Insert 
a copnia (e.g. De la Cruz becomes De^ la Cruz) 

Stripped out of the data arid disregarded In 
search. Headtnns must (should) be input 
without any diacrttics. (There are no diacritic 
on the terminal keyboard) 



Categdry 



Corporate Author 

Comnon name without 
subdivisions (United 
States) 



Full corporate name 
In AACR2 foraat 
(with parenthetical 
qualifier) 



LIMITS TESTiriG 
^ nf^«;FRVATinWS/RESULTS 



ILS 



Total retrieval hanpered by system Intolerance 
for Inconsistencies in form — U.S. U. S. and 
United States. 

Shows United States and United States plus sub- 
divisions alphabetically in index 

Not possible to specify all headinns containinq 
"United States" in corporate name search 



The addition of punctuation (period) to United 
States limits the index display to" ope headinn 
containing United States with a subd^tVision 
There is apparently a problem tn hanflltnti - 
internal and terminal punctuation in the 
indexinn of name entries 

Exact entry retrieves index^tries for corporate 
name and corporate name with subdivisions 

Truncation of corporate name by omittinn the 
parenthetical qualifier increases the retrieval 
sianificantly - . 



ERIC 
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CITE 



Total retrieval hamoered by system intolerance 
for inconsistencies in form ° 

Shows United States and United States plus sub- 
divisions alphabetically in index 

When s6archinn under series, conference, oroan. 
name (type S) the index display also includes 
"United States" aS a MeSH term (i.e. subject) 

Possible to specify aj_l headinns containina 
"United States .in ojrnan. name search, but it 
requires paninn thrbtSiJh all headings in the 
index first 

The addition of pWcfualTon "p?e^^^^ United 
States retrieves,^ index entries for corporate 
names and seriesNheadinns with United States 
and subHivistons \ 



Exact entry retrieves index entries for 
corporate name and corporate name with 
subdivisions . 

Truncation of corporate name by omittinn the 
oarenthetical qualifier 'increases retrijeval 
sianificantly ■ ' .i 
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Catcfory 



Title * 

Inconplete title 
(word which may be 
on stopword list 



LIMITS TESTIMG 

nn<;FRVATtfiN?i/REStJLTS 



ILS 



;/ 



Lona title beatnnlnp 
with generic word 
"pr oceedings" 



Truhcatlon of sane 
title 



Serial title with 
■ multiple editions: 
title chanoes 



ERIC 
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Numerous (49) titles beninnlncrKfth "Who's v»ho" ar 
retrieved with exact input of' those characters 



Title could not be retrieved as direct entry 
because it appears a§' title added entry on record 
and only the first occurrence of title field was 
Indexed in ILS. (Reeord was found by one tester 
by performinn term search) 



Cannot determine since title is not indexed 



Search retrieved 2 records- rone for the earlier 
serial title— but no information relatfnn this 
item to later title 

New serial title found by performing, a new search 

•on that title 



CITE 



Title cannot be retrieved If input with 
apostrophe 

Potential title match occurs when title searche 
wi thout apostrophe (2 ti tl es found , but^ only 
2nd one actually contained words "who's who"- 

Title found and was first in Ti St displayed 
amonn 323 possible items 



If truncate the title top early and oniit 
stnniflcant key words, cannot retrieve thjis 
title amonn thp first potential matches 
displayed 

448 records retrieved; 

American edition found by Itmitln? display to 
Enolish lannuape Hems 

Information about relattonshtp to new serial til 
was found in a note on earlier record. 
New serial title found by performing new search 
«on that' title. 
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Citf9ory 



Subjtcts 

MeSH heading with 
aoostrophe 
(Alzheimer's Ois. 




Combination of two 
single MeSH terms 
both with high 
postings 



HeSH heading qualified 
by cataloging sub- 
headinq 



LIMITS TESTING 

nRSFftVATTOMS/RgSIJLTS _; 



ILS 



MeSH heading not found in subject search 
Retrieval under term. "Alzheimer" and "Alzheimer's" 
as term search 

[etrieval under MeSH term "Dementia, Presenile" 
in subject search included citations for 
Alzheimer's Dis. (Alxheimer's His. is a 
minor desciptor, see under Dementia, Presenile) 



Relevant retrieval obtained by combination of 
terms in term search. Recall = 2 items 

Numerous steps required to select terms and then 
combine them 



Main headina qualified by subheadinn is indexed. 

9 items retrieved in subject search 
Order of input must be observed: subheadinn cannot 

be entered first because it is not a primary 

index term. 

Must input main headinn only and then select from 
list of subheadinns applied to that maip headino 



ERIC 



CITE 



MeSH headino found in subject search, terms 
displayed in subject search Included the 
singular form "Alzheimer" as a text word 

Presence or absence of apostrophe does not 
affect retrieval 



Relevant retrieval obtained by entering the 
search phrase exactly as suggested. Recall * 
151 items. * 

One search query required and selection from 
ranked terms 



Concept as reflected in the subheadinn (bibllo- 
oraohies) could onlv be approached throaoh the 
frjain headinn "Pi blionraphy" or as it appears 
in textword index. 
Subheadinns not indexed. _ ^ 

Order of input does not affect retrieval* Same 
23 i tems retri eved both ways . 



Category 



P unctuation 
Spell 1n(i variants 
tumor/tumour 



Truncation of term 
(steimlnci) 

Abort 

Abortion 

Aborting 



Hyphens and Spacing 
xrays 
X rays 
x-rays 



Word Order 
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LIMITS TESTiriG 

QBSERVATIOnS/R ESULTS 



C-6 



as 



Retrieval under both forms as Tonn as each one 

is separately searched. 
Retrieval is different for each form 
No acconnodation for soel lino variants 



Truncated form will display MeSH terms in the 
index or terms if entered as a term search, but 
selection of all terms requries many steps and 
user cannot consolidate retrieval of all items 
under the root word 

Best results were in usina truncated form as term 
search 



Each form results in retrieval, but the retrieval 
differs for each 

No acconnodation for variations in spacina of 
hyphenation 



Word order in subject search is critical since the 

entry must match MeSH term in index. 
Word order is not significant in term search 

since each term must be entered independently 
User cannot combine terms in initial term search 



CITE 



Same retrieval possible under either form since 
both forms are presented as ranked, search 
terms»no matter Which form is entered as searcl 
tern . 



Truncated form retrieves MeSH tetins and text 
words containinn the root word 



Retrieval is the same for term with or without 
hyphen. They are treated the same 
^Retrieval differs for form without space. 
Spacino is sitinificant 



Word order in subject search is not sianificant 
Retrieval is the iame either way 
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LIMITS TI:STIH6 

nR<;FRVATinH!;/l>ESIJLTS_ 



ILS 



Retrieval of series title with numberinn is ^ 
extremely tedious under option for series search 

Requires browsinp through multiple panes of index 
displays. 

Further complicated by system intolerance for 
variations ih spacinn and punctuation that occur 
because of cataloainn rule channes over the yearp 
Same series title may appear in widely separated 
parts of the index^ makinn collocatioa impossible 



CITE 



Same experiences as for ILS 

As series search it is extremely difficult to 

retrieve specific issue number. - 
Better to Search as subject in order to cjet 

textword retrieval and then scan all records 

retrieved for specific issue 



Retrieval under conference name search requires 
entry exactly as name appears in the index 
Recall = 8 items, but tester unsure about precis 

Retrieval under term search was more direct 

Cannot substitute "eiahth" for "8tK" 



ton 



Retrieval under subject search usjng textwords 
"Eiphth" or "8th" made no difference in subject 
search because retrieval was based on other 
textwords used In search 
"Eiqhth" and "8th" are sianificant if used in 
conference name search} entry must be exact 



Record found 



Record not retrieved. (Withdrawn items were 
excluded from ILS tape pulls as of 4/82) 
Other withdrawn items in ILS are displayed with 
the value "Withdrawn" in the call number 

Record found 

No call number displayed 
No other messaoe 



Record found 



Record found 

Call number is "Withdrawn" 
No other messaqe 



Record found " 
No call number displayed 
Messane: "This item not owned by NLM" 



T 
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Verification of Display Features 

Methodology: . 

The original requirements for. the display of elements ifl full and abbreviated 
formats listed the elements as "required", "necessary" or^"optional . Some ^ 
of the "required" elements were later modified based on the final »set ot 
requirements for test systems negotiated with the system developers. The 
presence of required and necessary elements in the ful-1 and abbreviated 
displays was verified by checking at least two records containing the^specified 
element in both, systems.. For the ILS. the MARC record display was defined as 
the full display and the standard display option given. 

Summary of Display Elements: ° 

Attachment A is a summary chart of the elements with the display requirements 
colded for Full (F) and Abbreviated (A) formats with checks, indicating the 
presence of the element under both systems^ 

The chart also indicates elements that were not available in either system, 
because they are hot present in any of the source CATLINE records (e.9.^uniform 
titles which have not yet been added to CATLINE). These el ement| should not 
present major problems for either system since the specifications are already, 
defined to both QATLINE and the CATLINE/MARC conversion programs. 

There are a number of elements included on the list iri Attachment A that are^ 
considered local-^data fi.elds (e.g. Dashed- On entry. Holdings) for which there 
is no corresponding field in the MARC communications format. These elements 
were not transmitted in the CATLINE records converted for« I LS J oading so they 
were not included in the verification testing of displays. The Shelving 
Location field is an exeption since a: special routine. was implemented by 
OCCS to convert this data for the ILS MARC tapes. 

The list below shows required and necessary elements that were not present 
in the full and abbreviated record displays for each system. The asterisk 
denotes those elements that are not present in the ILS records because they 
are local data fields not carried in the MARC format. 



Requi red Elements 



CITE 



ILS 



Full display ''"]\?' 

Conference Name • (all present) 

First/Last Issue 

Abbreviated display Abbreviated display 

(all present) . . 

^ Series Title 



Corporate Author/Title Seri 
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the compact paragraph format of^the biliographic record Is very 
difficult to read. Semicolons (;)are„used in CITE to separate 
discreet elements. However^ the semicolon Is also used 
standard ISBD punctuation Within elements, so another way of 
separating elements in CITE should be considered. 

The first occurrence of series fields (Series Titler Corporate/ 
Author title Series) is displayed before the Imprint in the ^ 
brief display, but prints after the Collation in the full S 
display. This seems unnecessarily confusing.; (See Example Z) 

Order of brief display: SE, IM, CO ^ , 

Order of full display: IM, CO, SE (this is standard cataloging 

order) 

The personal name entires (PN) are displayed after the Statement 
of Responsibility (AS) and this is also very confusing. 

the option for displaying the full MARC record is nor 
readily apparent to the user; This format is primarily 
for the use of catalogers and librarians accustomed to 
the MARC format. As is noted on the chart in Attachment A, 
some "necessary" elements for disfplay occur in coded form in 
the MARC format and are n'pt readily discernible to the 
non-librarian user. (See Example 3) If these elements 
retain their "necessary" status, they should be decoded and 
specifically identified for the user. 

In the brief display of the Corporate Name field, the 
subordinate body in the second subfield ($b in MARC) 
is not dispjayed. The corporate name fields are indexed^ 
through subfield b and it is present in the full MARC 
format, but has been dropped in the abbreviated display. 
(See Example 4) " 

In tlTe brief display, the title subfield orPersonal 
Names ($t of 700 tags) is not displayed. The name 
contained in a 700 tag that has a title subfield is 
displayed twice even though there is only one occurrence » 
of the 700 tag for that name. (See Example 5) 

The order of display for corporate and personal name 
added entries differs for serials and monographs. For. 
monographs, 'these fields precede the title, while for 
serials these added entries fpllow the subject headings 
at the end of the^record." This may. not be considered 
a significant differenice. but there should be some 
rationale for it. The ILS does, however, have a good 
feature in highlighting the title and differentiating . 

discreet elements. 
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Necessary Elements 

Full display 
Autograph Note 
Bibliography Note 
ISBN 

iBound With Note 

Necessary Elements— Full Display 

CITE ILS 

■Corporate Name 
Contents Note 
Date of Entry 
Dissertation Note 
Dashed- On Entry 
GMD 

History Note 
Holdings 
Item^ Type 
LC Card Number 
|.imited Use Note 
Last Revision Date 
Media 

Major Revision Date 
Open/Closed Indicator 
Personal Name as Subject 
Final Year 

Abbreviated, display 
Corporate Name 
General Notes 
Media 

MeSH Headings 

In addition to elements not present, there were some deficiencies or 
discrepancies noted in the display of elements that were outpfut in both 
of the systems. These are summarized here and illustrated in the attached 
examples of records printed from CITE and ILS. ^ 

CITE: 1. The first/Last Issue field for serials is displayed 
in the abbreviated display, but not in the full 
.display. This is probably an oversight rather than 
an 'indication of any problem in the output of this 
element. (See Example 1) 
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Full display 
♦Autograph Note 

Bibliography Note 
♦Dashed- On Entry 
GMD . 



Item Type 
Limited Use Note 
Media 

Major Revision Date 



Abbreviated display 
General Notes 
Media 
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ATTACHMENT A ; VERIFICATION OF DISPLAY 



-1- 
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R . RMuired for test system 
I : Necessary for operatdonal syste. 
0 » Optional 



* ™ 50 

T^^•.r^4>Afi fieia not 



ATTACHMENT A - VERIFICATION OF DISPLAY 



-2 



Elemen t N«ne 
|- Date of Entry 




R » Required for test system 
N - Necessary for operational system 



F • Full Record Display 
A - Abbr*viiited Record Display 

* Denotes lopal data field not _ 

« s:."::r.iii:aTt "^s:^^ ^ -ora. . 



0 m Optional 



Elew»nt Name 
Last Revision Date 



ATTACHMENt A - VERIFICATION OF DISPLAY 
^ Display/ ^tatos. 
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F • Full Record Dlaplay 
A • Abbreviated Record Display 
*Denates local data field not 



R ■ Required for test system 
N - Necessary for operational system 
0 ■ Optional 
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ATTACHMENT A - VERIFICATION OF DISPLAY 
Element W ane ^ ^^'f^'^/ States 



-4- 



** I Uniform Title 
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F - Full Record Di.pUy R - Required for test system ^ ^ 

A - Abbreviated Record Display N - Necessary for operational system 

* Denotes local data field not ^ „^ ° ' Optional 
' converted in the MARC format for ILS 

** Denotes field not; yet available on CATLINE source records" 53 
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EXAMPLE 1 



453.880 RECORDS SEfiRCHEP ' 

A3 ITEMS CONTAIN ONE OR MORE OF THE SEARCH TERMS 
I NONE 01- ItiE RECORDS HftlCH YUUK SEARCH QULSU ON EXACTin 

1/; Journal of alcohol and drug education.; v. 17, no. 2- 
: Lansing, Mich.; v.; Eng; M:1972 ' " 

CALL NUMBER; Wl J053^M; " 



k'inter 1972- 



1/; Journal of alcohol and drug education. ; Lansing, 
the Journal of alcohol education.; Issued 1972- 



of the Alcohol and Drug Problems Association of North fmerica.; Key title: 



Mich. ; V. ; Continues 
by the Education Section 



Journal of alcohol and drug education, ISSN 0090-1482.; 
Substance Abuse'; Alcoholism; Health Education 
CALL NUMBER; HI J0534M; 

CHOOSE THE nEM<S) IN WHICH YOU ARE MOST lNTERESTE|t» 

TYPE CHOICE NUMBERXS) OR NUMBER R.AMj5E<S> OR ALL 
or type X (to start new search)., or type STOP 



0351416; Eng; M:1^72; 



First/Last Issue field 



!Er|c 
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EXAMPLE 2 




I 



1/; Use of heteroploid and other cell substrate- fpr the product ion of 
bioligicaU; / editor, W. Hennessen.; Hennessen, H.; Developments in , 
hioloQicai standarditation t v. 50; Basel ; Net. York ll^iFiiFTTcTgHTTTTng , 

^' « THIS ITEM IS NOT YET OWNED BY NLM >> 
:2 " ■ 



1/; Use e>f >>eteroploid and dther cell substrates for th 
Tiologicals; v editor, W. +ienneEsen.; Hennessen, W.; Ba 
'Karger, :c1982. ; Developments in bioIogica K standard! zat 

ataloging in publication.; 8205062: EngrS:1982; Biolog 

inc 

<< THIS ITEM IS NOT YET OWNED BY NLM », 

CHOOSE THE ITEM<S) IN WHICH YOU ARE MOST INTERESTED 

TYPE CHOICE NUMBER<S) OR NUMBER RPNGE<S> OR ALL 
or type X (to start new search), or type STOP . 



acts: Cell 




Series 
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HI 



; EXAMPLE 3 ■ ■ 

Collins, C. H. , 
Society for Applied Bacteriology. 

Difeinfectantc. their M*^**^"^ evaluation of fif f ert 1 vg^egS / 

edited bv r -H. Collinfi 

London ; New York : Academic Press, 

xvi, 229 pi : iU. 8203467 



• , Disinfectants - congresses 
Sterilization - congresses 

Press <RETURN> to display next item, or _ - 

entSr /AU for author, /TI for title, /SU for subject, /TM for term search, 



CHOICE:' 



008 


r f 


010 


f t 


020 


9 9 


040 


1 t 


060 


9 r 


245 

• 


•00' 


1 260 


'0 ' 


1 300 




1 


• 4' 


1 500 


9 9 


1 

650 


9 2 • 


650 


' 2' 


. 700 


'10' 




'20' 











volume. I(e 



820211 



b 



1981 
q 1 



d 
r 0 



e „eri 
t 0 



f a 

V 0 



en 



a 
a 
a 
a 



a 
a 
a 



y - z 
80-41631 

012.1813800 . ■ - 

W?'"S0851F^no"l6 1981 a [QV 220 D6104 1979]^ 
Slsinfeltan^s! their use and evaluation of effectiveness / 

c edited by C.H. Collins ... let al.]. . 
London ; Kew York : b Academic Press, c 19B1. 

TKeSociety' for Applied Bacteriology technical series ; 

a "Thi4 volume includes contributions to the g'jt'f"" . 

Demonstration Meeting of Society for Applied . • 
Bacteriology which was held on 24 October 1979 at the 
Polytechnic of the South Bank, London. 

a Disinfectants x congresses 

a Steriiiiation x congtesses 

a Collins, C. H. , ^ , 

a Society for Applied Bacteriology. - 

a820211s198t enka ^ ^ nnoi7 K i i 

a 00958._i(rn c..a„: d.(©.....e...2 f..2 J. 00?.1.7........h „...i...i... 
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_ A-i^ 

iXAMPLE 4 I Lr & 

Mv Journal o f ;^lcohol anH drug g<JapatiPT»t . • 

J0534M V. 17. no. 2- winter 1972- <-»^Jal 

Lansing, Hich. iaor-|«l 
V. €351416 

' Substance Abuse - prevention « control - periodicals 

Alcoholism - pre^vention A control - periodicals 

Health Education periodicals r id«,>k fi-oriVa 
.Alcohol and Drug Problens Association of North flmerica. 

Press <RETURN> to display next item, or^ _ . +pr« search 

enter /AU for author, /TI for title, /SU for subject, /TM for term search. 

CHOICE: 



'730305 b e c 1972 d 9999 e xx f u g u h i 
j k n n P q 0 r u s u t a v 



HARC FORMAT OF RECORD: 

V 

. 008 • • a ?...u. . — ^ - ^ ^ ^ ^ 

I a w 0 X eng y z d 

I 022 • • a 0090-1482 

! 040 • • a DNLM c DNLM 

I 060 • • a W1 J0534M ' ^ * 

i 222 '00' a Journal of alcohol and drug education 

I 245 '00' a Journal of alcohol and drug education. 

260 'OO' a Lansing, Hich. 

300 • ' a . V. ^ 

350 ' ' a 4.00 (journal price) ,0,0 

Ul 'I' I l4sied V972?" by'Ihe Wuciiion Section of the Alcohol 

550 0 a ^^JI^^^JJ^-'probleBS Association of North America. 

€50 '2' a Substance Abuse x prevention 4 .control «^5®;:»?°*^**® 

650 ' 2' a Alcoholism k prevention a control x periodicals 

€50 " 2* a Health Education * periodicals . ft-^-ica b 

7fo '20! a Alcohol and Drug Problems Association of North Rmerica. v 

- - Education Section. i -- 

l~780 ' '00/ ^ Journal of alcohol education 

1 XCL • ' a 730305c19729999xx uu p o^''''?^S®"^« on;^53 hi 1 

i YYY • ' a 00916 be ca „d„B e? A^:..~Si.M^ — l- 
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. EXAMPiE 5 

M2 Darwin, Charles Robert 

too ^ Barrett, Paul H. 

0228c" Barrett, Paul H. 

1974 - - 



\L.S 



DarMiTi on fH^r.r> ncvrhol n i rf l] ^t V^^^ ^ cientif c creativity, 
iifip^i^f^'k^^ n^^l j^nok^ annotated by Paul Ht 



Barrett. 

fJew Yorif, Dutton, 1974. ..^.^oo 
XXV, 495 p. lUus., ports. 75005B9 



Creativeness 
Evolution - history 



«f ai?h^jrJ?ll ?orti;!e: ?SU fo. subject, /TH for te.. sea.eK. 
I CHOICE; 

J ■ - ^ ■ . 



MARC FORMAT OF RECORD: 

OOe • "• .7502.5 b. c .97. , , " 

010 • • a 76-122778 
020 ' ' a 0525088776 
040 ' ' a DNLM c DNLM . 

ol^wii's e^rly and unpublished notebooks, transcribed 
and annotated by Paul H. Barrett. 
250 ' ' a listed.] 

260 '0' a New York, b Dutton, c19/l. 
300 • • a XXV, 495 p. b illus., ports. 
? 350 ' ' a 20.00 
• ^50 '2' ^ Cteativeness 
i 650 ' 2' a evolution x h.!..-:* - 

notebook s> . 

XCL • . a 750215S1974 us ac p^°T2^"^W9 h i 

YYY ' ' a OOB60 b n c a dm e 2 f 2 g OU^^fS n i 

J 4 k 5 m 0 
111 ■' ' b 7500589 c 811030 

f XiS^'^UthSrJn «U fo. subiect, /TH for ter. search. 
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APPENDIX B. 

User Acceptance - Results of Data Analyses Across Methods 
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Users of the computer catalog repr esent a b road cross 

^cuiJiuii ui I KUl LbblUWAL° 





C/TT 










09) 


(37) 




r3v) 




or) 


1 




























M 


(^■>) 



























o 60 
ERIC 



JOSX..IlSPfi&:J9£..^S»<5 ^QM P' J T E ri CA T AL OG An C I MrRCQ fJ il - yjT - srv ri-R ^ 



TIME VISITORS TO THE LIBRARY. 

























» 
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Host users qf the computer- catalog come with subject- 



related information. 





























Vo/ 7 0 


> & " 
















> 
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iloST U3ER3 or VAL COMPUTER CATALOG ARF I OOKlNg F OP 
BOOKS ON A TOPIC, " „ 



rcr 







u ■ ■ - 








■ ft 
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-^M&sT-ysei^ OF th e C omputer catalog search by subject or 

TOPIC. 

V ■ ■ ' ■ 





33* </ 






3^.1 
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CITE USERS PREFER THE COMPUTER CATALOG TO THE CARD. CATALOG 
IN HIGHER NUMBERS, 





CO) 


C33> 












- A/-r 
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Among PATROWS/WHO HAVE usEp.COWCAT/ preference for the 

COMPUTE!? CATALOG 15 EQUlVALErn" FOR CITE AND ILS USERS, 























. • • • . ^ 


- •> 




A- 


r 



\ 



•T . 
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More information is found by users of CITE. 







C 












• 
















^"^ — — ~ — ' — — 
























» ■ ■ 









■ u^Tt?tc7' SeAfi^H^I^S (a)- J>r) R>vJAni jv»(J>2fc ;a//=6 £>aJ,,, ^ 



CtTtT 



Cv) 



7r 5/0 



67 



"Satisfaction WITH SEARCH RESULTS IS HIGHER AMONG USERS 
OP CITE. A t 



/tYivy u^k) J AT" / S firry of^l 



< ,^ X ^ U.f.3 



C/T^ (.1/) j'x % 



0 
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OVERALLSATISFACTION WITH THE COMPUTER CATALOG IS HIGHER 
AMONG USERS OF CITE. 



• ■ ■ 


c 1 rtr 


















(-7) 























FfifF. or) :L:r 



^0 Cjt) 
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Professional role/occupation is unrelated to satisfaction 
WITH CITE and ILS. 

- A^Lf^ /ecru J' Stnt. Sun.\^tr^ (^^^od) 

- ^Ay^pLl^ ^iTAYlCH ^rKfl- (aJ ^ ^) 



/ 



CITE IS PREFERRED FOR SUBJECT SEARCHING, 









> 














♦ 














/J' f 












< 







S/s»M(?Lt^ S<>-^cH- A^=:\o rr^vPP 

Jus ' aJ, b. 



er|c 



Moir S/^T»r;v^7<w7 C/0''<=>'^'? ^').-r% ^ /j^^u 
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5, 



CITE IS PREFERRED FOR TITLE SEARCHING. 















^ » ffc^' r'^* fcgi'^ L>OfTof<MJC 


Of) 












C « / 


£ . \ 






3^.3 


^CfTH^m A til 














.(/^; 










i 














SAT\fi.r e^vpr V) rr>^/^^ 

























(r) % 
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No CLEAR-CUT PREFERENCE FOR CITE/I LS IN PERFORMING OTHER 
TYPES OF KNOWN ITEM SEARCHES. 

0 




4 
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adUMiiiiiiiiiiiii 



CITE AND ILS DISPLAYS ARg/ OVERALL/ EQUALLY ACCEPTABLE 



TO USERS 1 










''scfirrJfifMj- rmj ^»AJc^ 
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CITE INSTRUCTIONS. PROMPTS, "HELP" MESSAGES TaRE SOMEWHAT 
MORE EFFECTIVE. 



■ \ 
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APPENDIX e. 
CLR User Questionnaire (NLM Version) 
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Gouncil on Library Resources 
COMPUTER CATALOG STUDY 
User Questionnaire x 



The library is conducting a study of its computer catalog to improve it. This question- 
naire is a way to communicate your views. It shou,ld take you only about 1 5 minutes 
to complete. Your responses are confidential. Please do not write your name anywhere 
on the questionnaire. Thank you. 



MARKING INSTRUCTIONS 








• USE A NO. 2 PENCrL only; 




• FILL THE CIRCLES COMPLETELY. 








• BE SURE TO ERASE CLEANLYlVNY 




MARKS YOU WISH TO CHANGE. 








• MAKS NO SI RAY MARKS ON THIS 




QUESTIONNAIRE. 






o 




o 








o,. 



ERIC 



OO 



ooBBOBooooooooo 22989 

DO NOT WRITE IN THIS AREA 



NCS Trant-Optic 0S-1S301:321 
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mmm 



^ PART 1: ABOUT YOUR MOST RECENT SEARCH 

INStBUCTIONS; PI«Me answer t^ese questions about the computer catalog search you just 
completed*. 



1.;; I came to this comptjtlr search with: 
. (Mark ALL that sp6Iv\ 

a. A complete author's name O 

b. Part of an author's name O 

c. A complete title O 

Part of a title O 

p. A topic word or words , *.0 

f. A subject heading or headings O 

g A complete call number , . O 

h. Part of.a call number i O 



2t By searching this computer catalog I was trying 
to find: 

(Mark ALL that apply) 

a. A specific book, journal or magazine O 

. ,b Books, journals or magazines on a topic or subject O 

c. Books by a specific author ^ O 

d. Information such as publisher, date, spelling 

of^a name, etc O 

e; If a book that I know the library has is available # _ 
for my use ^ .• ?r.O 

f. Another library that has a book, journal or 

magazine that I want — O 



i- ■ ■ ? 

3. I searched for what I wanted by: 

(Mark ALL that apply) 

1 

a. A complete author's name ....o 

b. Part of an author's name » .O 

c. A complete title .t' O 

d. Part of a^itle O 

e. A topic word or words . .t * — O 

f. A subject heading or headings .O 

g. A complete call number O 

h. Part of a call number O 

\ ■ ■ 
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4.. I need thia infoPriiilion for: 

(Mark Aj^ that apply) ^ 

a. Recreational uses — O 

b. Making or fixing sornething O 

c. My work or job ,^ O 

d. Personal Interest ^ . . . . » O 

e. A hobby O 

f. Class or course reading .O 

g. A course paper or report ^ . O 

h. A thesis or dissertation i O 

i. Writing for publication O 

j. Teaching or planning a course* ^ O 

k. Keeping up on a topic or subject O 



5. In thia computer search I found: 
(Mark ONE only) 

a. More than I was looking for o 

b. All that I was looking for o 

c. Some of whart was looking for O 

d. Nothing I was looking for ' ..::.......0 

6. In ^relation to what I was looking for« this computsr . 
aearch was: 

(Mark ONE only) 

a. Very satisfactory i i . . O 

b. Somewhat satisfactory O 

c. Somewhat unsatisfactory O 

d . Very u nsat isf actory O 

7. 1 came across things of interest other than^what I was 

looking for: 

a. YES .O 

b. NO O 



B. I got help in doing this computer catalog starch from: 
(Mark ALL that apply) 

a. Printed material or signs O 

b. Instructions on the terminal acreen .O 

c. Library tfiaff member \ — ......... ^. ... O 

d. Person nearby ;. ^ 'i O 

e. j did not get help . , O 
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9. My overati or general attitude toward the computer 
catalog is: 
(Mark ONE only) 

• Very favorable 

to. Somewhat favorable 

c Somewhat unfavorable . . ; 

d. Wry unfavorable — 



10. Compared to the card. book, or microfiche catalog 
in this library, the computer catalog is: 
(Mark ONE only) 



O 
O 
O 
O 



a. Better 

b. About the same 

c. Worse : 

d. Can't decide . . . 



o 
o 
o 
o 



PART*e: YOUR EXPERIENCE WITH COMPUTER CATALOG FEATURES 
IIMSTRUCTIOiMS: Mark the single column for each question that corresponds most closely to how 
you feel. If the statement does not apply to your experience at the computer catalog, mark the 
column. "Does Not A'pply". " 



STRONOLV 
AGREE 



AGREE 



NEITHCR 
AOR^E NOR 
OlSAQREE 



OlSAGREE 



StRONGLY 
OlSAGREE 



16 Remembering commands in the middle of the search is easy . 

17 Finding the correct subject term is difficult 

18 Scanning through a long display (forward or backward) is easy 

19 Increasing the result when too little is retrieved is difficult . 
20. Reducing the result when too much is retrieved is easy ....... 



1 A computer search by title is difficult 

2. A computer search by author is easy 

3 A computer search by subject is difficult 

4 A computer search by call number is easy 

5. A computer search by combined author /title is difficult 



2V Understanding explanations on the screen is difficult 

22 Using codes or atbbreviations for searching is easy 

23 Abbreviations on the screen are easy to understand 

24 Locating call numbers on the screen is'difficult 

25 Searching with a short form of a name or a word (truncation) is easy 



26 Using logical terms like AND. OR. NOT is difficult 

27 Remembering the ^xact sequence or order of commands is easy 

28 Understanding the initial instructions on the screen is difficult 

29. Understanding the display for a single book, journal or magazine is easy 

30 Understanding the display that shows more than a single book. 

journal or magazine is difficult • • • • 



31 Interrupting or stopping the display of information is easy 

32. Typing in exact spelling, initials, spaces and hyphens is difficult to do . . 

33 Knowing what is included in the computer catalog is easy to remember 

34 The order in which items are displayed is easy to understartd * 

35 Displayed.B3«ssages are too long , . • •; •• • — 
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O 

o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 



STRONGLY 
AGREE 



AGREE 



NEITHER 
AGREE NOR 
OlSAGREE 



o 
o 
o 
o 



o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 



DISAGREE 



STRONGLY 
DISAGREE 



O 

o 
o 
o 

o 



o 
o 
o 
o 
o 



o 
o 
o 
o 



o 
o 
o 
o 



o 
o 
o 
o 
o 



o 
o 
o 
o 
o 
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36. Selecting from e list of choices takes too much time 

3^7. EntennQ commands when 1 want to during the search process is 
difficult ......♦.* 



38. The rate at which the -computer responds ts too slow 

39. The availability of signs and brochures is ade<|uate . . 

40. Signs and bcochures are not very useful ...... .1 

41 The staff advice is often rwt helpful ^ 

42. It IS fJafd to find a free terminal 



T" 



STRONGLY 

AoacK 


AGREI 


AlEflTNtfl 


DiiAcaiE 


smoMOiv 
otsAoacc 


NOT 
APflV 


0 


0- 


0 


. 0 


0 




o 


o 




n 




A 

n 


o 




o 


o 


o 


n 

w 


o 


o 


o 


o 


n 


n 


o 


o 


o 


o 






o 


o 


o 


o 




n 


0 


0 


o 


o 

x^ 


O 

x^ 


o 



YOU ARE MORE THAN HALF - WAY DONE 




PART 3: IMPROVING THE COMPUTER CATALOG 
Select the response oi responses that best reflect your vievys about changes 



IIMSTRUCTIONS 

that should be made in the computer catalog 



43. When I use 'the computer catalog terminal: 
(Mark ^ES or filQ) 



a The keyboard is confusing to use 

b. There is too much glare on the screen . . . . 

c. The letters and numbers are easy to read . 

d The lighting around the terminal is too 
bright » 

e There is enough writing space at the 

terminal ,. . 

I. Nearby hoise IS dtsstracting ., 

0 The terminal table i^oahi^ or too low . . 
■ h. The printer is easy to use 



YES 



O 
O 

P 

O 
O 
O 
O 



NO 

-& 

o 
o 

o 

o 
o 
o 
o 



. \ ■ ^ . •■ • 

4. Select up to FOUR addjtional features you would 
like this computer catalog to have': 

a. Providing, step by step instructions > O 

b. Searching by any word or words in a title O 

c Searching by any word or words in a subject 

heading O 

d. Limiting search results by date of publication * O 

e. Limiting search results by language O 

f. Ability to search by journal title abbreviations O 

fl Ability to change the order in which items are 

displayed > O 

h. Ability to view a list of words related to my 

search words ., O 

I. Ability to search for illustrations and 

bibliographies , . . O 

j Ability to search by call number . /, O 

k. Ability to print search results 



--r -"^V^. 



o 

....O 

n, Ability to tell where a. book is located in the ^ 
library O 

0 None O 



I. Ability to search a book's table bf contents. 
Summary or index 

m. Ability to know if a book is checked out ... 



45. Select up to FQUFI computer catalog service improve* 

ments you would like the library to make: 

* * .' " . 

a. More terminals , O 

b. Terminals at locations other than near the card catalog . O 

c. Terrninals at places other than library buildings - O 

d. A chart of commands posted at the terminal O 

e. A manual or brochure at the terminal O 

f. An instruction manual for purchase O 

g. Training sessions ^. 10 

h. Slide /tape /cassette training program ......... O 

i. None O 



4@. Select up to FOUR kinds of material you vvould like to^ 
see added to the computer catalog: 

a. Dissertations . , . , O 

b. Motion picture films ..i.^.t O 

c. Government publications O ^ 

d. Journal or magazine titles, O 

e Maps ...... r .* O 

f. Mahuscripts O 

g. Music scores O 

h. Nevvspapers O 

I Phonograph records or tapes O 

j. Technical reports * O 

k. More of the library's older books O 

\. N<5ne " *. O 

Other O*^ 

47, BRIEFLY DESCRIBE ANY OTHER PROBLEMS WITH 
THIS COMPUTER CATALOG OR CHANGES YOU 
WOULD LIKE MADE TO IT: , Q 
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PART 4: ABOUT YOURSELF " C-'^ 

INSTRUCTIONS: Ycur rMponses are cohfidcntiai. Please do not write your name anywhera on 
thif questionnaire. 



./ 



48. J coma to this library: 

■ Daily' "1 O 

b. VVMkly ;.....^..v^ ...,0 

c. Monthiy ..Q 

d About (our timts a ytar O 

• About onca a yair ^ ..... O 

' f. Nol bafore today O 



49. I usa this computsr cataloy: 

• . Evif y library visit , O ' 

b. Almost avary vi$it ...O 

c Occasionally i v , .« O 

dlRaraly Q 

ft. Not before today y . . , O 



50. I usa this'library's bpok, card or microfilm catalog: 

a. Evecyvisit ; O* 

b. Almost fevery visit , ,...;,,/...... O 

e. Occationally \, ....O 

<i. Rarely O 

a Never • O 

51. I u%h a computer system o^har than the library^i 
computer catalog: ' 

/' ' . *' I' 

a. Daily . . .\ ....... ... . . . « /:.... O 

b. Weekly .....v..;........:/....'...... .O 

■ c. Monthly' ^. ...... .1 .. ^. ... , ..... O 

d. About four times a" year \,. O 

^ e. About once a year ...... O 

f. Nevpr ' : O 

/ ■ ■ * ■ 



52.. I first heird about this computer catalog^from/ 
(Mark ONE onM v • ^ 

a. Noticing a terrhtnal ia^the library O 

' b. Library tduK orientation pr jJemonstration ^ . , . .O 

c An article or wrineh annour>cement , . .O 

d. A course instructor .v w O 

e. A friend or f aHCj ly^member ..... O 

i Librarystaff ...r^ ..O 

O ■ . \ • . . ^ ■ ' * ' . • ^ ^ 

ERLC - ^ . . ' 



/ 



'53. I learned how to use this cortiputer cetelog: 
' (MerkAjyt that apply) < < 

a. From airieild or someone at a nearby tern^inal .". .O 

b. Using printed instructions .Q 

c. Using instructions on the terminal screen ^ , O - 

d. From the library staff ; . . .Q 

e. From a library course or orientation . . » ^ * . , .O » 

f. From a slide/tepe/cassette program ...O 

g. By myself without any hel^ , , , Q 

>. ■ • - * ' 

64. My age group is: 

a. T4 a.nd under ......^ .<..... .....,.?.0 

b. 15 . 19years .Q 

c. 20- 24 years J ! O 

d. 25- 34 years Q 

e. 35 - 44 years O 

f45 .S4 years ...Q 

' g. 55 - 64 years , .....O 

h. 65 andtjyer .^U-V'^a- • ....O 

■ ■ , .^^{^* * 

, . ■ ^ - m^^ . , ' ■ 

J55. lam;. _ ■ jr^^ , -'^ 

a. Female--. ".Ta O 

b. Male ... ...... :;...vl. ...... .0 

. *. ^ ' ■ "■ ■" 

56' Merk your current or highest^educetionel lefvel: 
(Merk ONE onh^) 

a. Grade School or Elementary SchopI . W 

b. High Schooler Secoridary School O 

c. Son.e College or UniversitV-- . i , , O 

d. College or^ University Graduate , ; O 

If you ete not completing this quei^tionnaire at a dollege or 
university/pleese stop here. Thenk you. . 

" ' " ' ' ' I j - - 1 - 

• * . I o 

If you are completing ^this ques]tionneire et a college or uni* 
* yersity, pleese continue. 



57: The cetegory thet best describes nrty ecedemic eree is: 
(Mark ONE only) ^ 

a. Arts and Humanities ..O 

b. Physical/Biological Sciences . . . . O 

c. Social Sciences ..\ .»0 

d. Business/Management ........ ^ Q 

e. Education ........v.. O 

f Engineering .. .\ ....... . ^ . I.. . ". . . . O 

g. Medical/Health Sciences O 

b.Uw O 

I Major not declared . ^ \j 

j. Interdisciplinary - . * — i , Q 

•• SI ■ ■ 



• SUPPLEMENTARY QUESTIONS 

Instructions: Questions 57 through 61 are to be answered directly on this page. 
For each, fill in the "0" correisponding to the response you choose. 



57. 



a. 
.b. 
c. 
d. 
e. 
f. 

9- 
h. 
i. 
j. 



58. 



a. 
b. 
c. 
d. 
e. 
f. 

9- 
h. 
i . 



59. 



a. 
b. 

c. 
d. 

f. 

9- 



The term below that best- describes 
my subject area or specialty is: 

Allied Health ..... 0 

Biological Science 0 

Dentistry 0 

Library or Information Science .. 0 

Medicine ; 0 

Nursing 0 

Physical Science 0 

Social -Behavioral Science 0 

Veterinary Medicine •« • 0 

Othef^i.e., none of the above) T O 



The^Hepm below that best describes 
my professional role is: 

Health Care ^Practitioner ........ 0 

Health Professions Educator ..... 0 

Researcher^ Biomedical 0 

Researcher » Other ; 0 

Student, Health Professions ..... 0 

Student, Graduate .1.........'.* .. 0 

Student, Undergraduate 0 

' Other Professional 0 

Other (i.e., none of the abov6) . 0 



first and primary use of the 
information^ obtained here today 
will be in: , 

Courses (classes) I am taking ... 0 
"Keep'^ng up" with a topic or. 

field 0 

^ Patient care ; 0 

Preparing an article or other ^ 

publication .^5^.....*.... 0 

Planning or conducting research . 0 

l^lanning or teaching a course 0 

6ther (i.e., none of the above) . 0 



60. 

a. 
b. 
c. 
d. 
e. 
f. 
9- 

h. 

i. 



61. 



a. 
b. 
c. 
d. 
e. 



My' primary place of work is: 

A private practice 0 

A college or university 0 

A hospital 0 

A' private company or business ...... 0 

The National Library of Medicine ... 0 

The-National Institutes of Health .. 0 
A federal agency other than the 

NLM or NIH 0 

In no formal organization (i.e., 

unaffiliated) 0 

Other (i.e. , none of the above) .... 0 



CofDpared to COMCAT (the microfilm 
catalog in this library), the computer 
catalog is: 

Better •• 0 

About the same 0 

Worse 0 

I can't decide •• 0 

No inl&ormation — I have not used 
COMCAT 0 



Thank you for participating jn 
THIS STUDY. "This completes the , 

QUESTIONNAIRE. PlEASE TURN IT. 
IN AS INSTRUCTED.' ^ ^ , 
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APPENDIX D. 
CLR Non-User Questionnaire (NLM Version) 




^ % ' , • . ■ " ■ . , .,♦ ... .*« .. .. V a- ■■ 



Council on Library Resources , 
COMPUTER CATALOG STUDY 

Questionnaire 



The library has a computer catalog and not everyone has had a chance to use it yet. If you have 
NEVER USED THE COMPUTER CATALOG you can contribute to the quality of lib«ry services 
by completing tfirs questionnaire. It takes about five minutes. Your responses are^onfidential. 
Please do not write your name anywhere on the questionnaire. Thank you. 



• ' * > 


■ C 




MARKING INSTRUCTIONS 

• USE A NO. 2 PENCIL ONLY. ° 

• FILL THE CIRCLES COMPLETELY. 


/// 

/«y/ 




^ • BE SURE TO ERASE CLEANLY.ANY 
MARKS YOU WISH TO CHANpE. 






• MAKE NO STRAY TVIARKS ON THIS 
QUESTIONNAIRE. 


(J 




a 

t » 

. • 0 


o 
o 
o 
o 


„ * , • *^ ■ 




PART 1: WHAT YOl^ THINK ABOUT 
THE COMPUTER CATALOG 

INSTRUCTIONS: PieaM mark the response 
that be$t describes how you view a computer 
catalog. 



1 ; I have not used the computer catalog up to now 
because: 

(Mark ALL that apply) 

a. t do not tike to uss computers O 

b. 1 did not know there was a computer catalog O 

c. I do no^know Where it is O 

d. I have not had time to learn to use it O 

e. 1 hsve not taken training sessions on how to use it O* 

'(. There has not been any staff^at the terminals to ^ 
assist me in using it O 

0- The terminals were all in use whifn I wanted to ^ 
use it O 

h. I have not needed to use any library catalog ^ 
recently O 

i. The card catalog is easier to use O 

j- The card catalog contains more of the information ^ 
I need ^ O 

k. i am a visitor or infrequent user of this library O 

■ ' ■ f 

2. How much time do you think it takes to learn to use 
the computer catalog? ( 

a. A day or more « 0 

b. Between 1./2 of a day and a day O 

p. Between an hour and 1/2 of a day — O 

d. Between 30 minutes and an hour .\ O 

e. Between 15 minutes and 30 minutes O 

f. 1 5 minutes or less O 



3. How difficult or easy do you think it would be to learn 
to use the computer catalog? 

: \ - 

a. Very difficult O 

b. Somewhat difficult O 

c. Somewhat easy*^ — O 

d. Very easy . . O 



^4. My overall or general attitude toward the computer 
catalog is: 

a. Very favorable O 

b. Somewhat favorable O 

c. Somewhat unfavorable ^. O 

d. Very unfavorable^. O 



5. How likely are you to use the computer catalog in the 
future? ^ 

a. Very likely \ O 

b. Somewhat likely /• ; O 

c. Somewhat unlikely O 

d. Very unlikely O 



6. Compared to the card, book, or microfiche catalog 
in this library the computer catalog is: 
(Mark QNE only) 

a. Better O 

b. About the same O 

c. Wors6 O 

d. Can't decide O 



PART 2: ABOUT YOURSELF 

INSTRUCTIONS: Your responses are coh- 
fidential. Please do not write your name* 
anywhere on this questionnaire. 



7. I come to this library: 

a. Daily . . , . ! : O 

b. Weekly O 

c. Monthly O 

d. About fQUr times a year \ o 

e. About once a year O 

f. Not before today ; O 



8. I use this library's book, card or microfilm catalog: 

a. Every visit • • • • • O 

b. Almost every visit .0 

c. Occasionally O 

' d. Rfrely \ O 

e. Not before today O 

1^ 



I ust • compttttr $y$t«m othtr than tht library's 
computar catalog: 

a. Daily ; O 

b. Wokly O 

C Monthly O. 

d. About four times a year , O 

e About once a year , O 

. f. Never O 

10. My age group Is: 

a 14 and under , O 

b. 15- 19 years O 

c 20 - ^4 years O 

d. 25 - 34 years O 

35 - 44 years O 

f 45 • 54 years O 

g 55 - 64 years O 

h. 65 and over O 

11. lam: 

a. Female O 

b. Male 0 



12. Mark your current or highest educationallevei: 
(Mark ONg only) 

a. Grade School or Elementary School O 

b. High $chooi or Secondary School O 

c. Some College or University O 

d. College or University Graduate O 



if you §£6; not completing this questionnaire at a college 
or university, please stop here. Thank you. 



If you are completing this questionnaire at a college or 
university, please continue. 



13. The category that best describes my academic 
area Is: (Mark ONE only) 

a. Arts and Humanities — . . O 

b. Physical/Biological Sciences O 

c. Social Sciences ...i , O 

' d. Business/Management O 

e. Education Q 

f. Engineering .* ^ O 

0. Medical/Health Sciences O 

h. Law O 

1. Major not declared *. O 

j. Jntefdisc iplinary ^ Q 



14. Tha main focus of my acadtmic work at the pfMam 
time is: 

(Mark A^L that apply) 

a. Course Work O 

, b. Teaching O- 

c. Research * ........ ..O 



15. My prasem affiliation with this college or unhraraity la: 

a. Freshman/Sophomore O 

b. Junior/Senior O 

c. Graduate - masters level O 

d. Graduate - doctoral level :. . O 

e. Graduate - professional ichooi O 

f. Faculty .....Q 

g. Staff O 

h. Other , , O 



Thank you for participating in this study of the 
computer catalog. This completes the question- 
naire. Please return it. 



SUPPLEMENTARY QUESTIONNAIRE ITEMS 




16. ooooooooooooooo 

17. ooooooooooooooo 

18. ooooooooooooooo 

19. ooooooooooooooo 
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SUPPLEMENTARY QUESTIONS 

Instructions: Questions 13 through 16 are to be answered directly on this page. 
For each, fill in the "0" corresponding to the response you choose. 



13. 



a. 
b. 
c. 
d. 
e. 
f. 

g- 

h. 
1 . 
j. 



The term below that best describes I 15. 
my subject area or specialty is: 

Allied Health ................... 0 

Biological Science 0 I a. 

Dentistry 0 ' ^' 

Library or Information Science ..0 

Medicine 0 I 5* 

Nursing ••• J ' 

Physical Science 0 

Social -Behavioral Science... 0 e. 

Veterinary Medicine 0 f. 

Other (i.e., none of the above) . 0 I g 



My first and primary use of the 
information obtained here today will 
be in: 

Courses (classes) I am taking .. 0 

"Keeping up" with a topic or 

field 0 

Patient cariB .■ 0 

Preparing an article or other 

publication 0 

Planning or conducting research .... 0 

Planning or teaching a course ...... 0 

Other (i.e., none of the above) .... 0 



14. The term below that best describes 
my professional role'is: 

a. Health Care Practitioner 0 

b. Health Professions Educator 0 

c. Researcher j Biomedical 0 

d. Researcher, Other 0 

e. Student, Health Professions 0 

f. Student, Graduate 0 

g. Student, Undergraduate 0 

h. Other Professional :***;** « 

i. Other (i.e., none of the -above) . u 



16. My primary place of work is: 

a. A private practice 0 

b. A college or university [J 

c. A hospital 0 

d. A private company or business ...... u 

e. The National Library of Medicine ... 0 

f. The National Institutes of Health .. 0 

g. A federal agency other than the 

M.M or NIH *•• " 

h. In no formal organization u.e., 
unaffiliated) ^ 

i. Other (i.e., none of ttie above) u 



Thank you for participating in this 'study.' 
This completes the questionnaire. 
' Please turn it in as- instructed. 
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APPENDIX E. 
\ ■ ■ ■ . 

Sample Search Experiment - Search Questions (Odd/Even Versions) 
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ODD 



SAMPLE SEARCH EXPERIMENT , 

A. Personal Author Search 

Print a bibliography of recent (1974- present) publications by:. . 

Harold I. Kaplan (CHE) 
Charles Horace Gray ULi; 

STOP - CHECK-MARK RELEVANT RECORDS - ANSWER QUESTIONS 



B. Corporate Author Search 

Find books published by a particular organization.' The name is 
not complete as presented: 

Harvard Center for Medical Care 

Cente** for Ulcer Education (CITE) 

STOP — CHECK-MARK RELEVANT RECORDS - ANSWER QUESTIONS 



C. Title Search 

Find the following titles. Part of the title is: 

Host-Virus Interaction . nllV 
Activity in Proteins UL5; 

STOP - CHECK-MARK RELEVANT RECORDS - ANSWER QUESTIONS 



D. Series Search ' 

Find the books which are part of a series. The incomplete name of 

the series is: ^ 7 . . 

Laboratory and Research Methods !1ttc\ 

Wiley Series in Probability and vCITe; 
Mathematical Statistics 

STOP - CHECK-MARK RELEVANT RECORDS - ANSWER QUESTIONS 
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E. Conference Search 

Find the records from the following conferences: 

Conference on Ambulatory Monitoring 
Conference on Recombinant DNA 



(CITE) 
(ILS) 



STOP CHECK-MARK RELEVANT RECORDS - ANSWER QUESTIONS 



F. Sub, 7ect Searches 

Find books on the following subjects: 

Smoking and Lung Neoplasms 
Cerebrovascular Disorders and 
Contraceptives, Oral 



(ILS)^ 
(CITE) 



Search the following subjects: 

idxic Shock Syndrome \ \ulV 

Percutaneous Transluminal Angioplasty {iL:>) 

STOP - CHECK-MARK RELEVANT RECORDS - ANSWER QUESTIONS 
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EVEN 



SAMPLE SEARCH EXPERIMENT 

A. Personal Author Search 

Print a bibliography of recent (1974- present) publications by: 

Harold I. Kaptan ULS) 
Charles Horace Gray CCITE) 

STOP CHECK-MARK RELEVANT RECORDS — ANSWER QUESTIONS 



B. Corporate Author Search " , 

Find books published by a particular organization. The name is 
not complete as presented: 

Harvard Center for Medical Care (CITE) 
Center for Ulcer Education (ILS) 

STOP— CHECK-MARK RELEVANT RECORDS - ANSWER QUESTIONS. 



C. Title Search 

Find the following titles. Part of the title is: 

Host-Virus Interaction \^^r\ 
Activity in Proteins (CITE) 

STOP - CHECK-MARK RELEVANT RECORDS --^ ANSWER QUESTIONS 



D. Ser^ies Search 

Find the books which are part of a series. The incomplete name of 
the seri.es is: " 

/ , Laboratory and Research Methods S^^^x^- 

. / Wiley Series in Probability and (ILS) 

'/ Mathematical Statistics ^ 

STOP - CHECK-MARK RELEVANT 'RECORDS - ANSWER QUESTIONS 



^'3 



E. Conference Search ' . " 

Find the rfecords from the following conferencfes: 

Conference, on. Ambulatory Monitoring (ILS)-. 
Conference on Recombinant DNA^ ' (CITE) 

STOP— CHECK-MARK RELEVANT RECORDS ^- ANSWER QUESTIONS 



Sub.iect SearcRes . . 

Find books on the foTlowing subjects: 

SmoRing knd Lung Neoplasms > (^^^x^ 
Ce^ebrovasculari)isorders and (ILS) 
Contraceptives, Oral 



Search the following subjects: • 

Toxic Shock Syndrome Iit5ix 
Percutaneous Transluminal Angioplasty (CITE) 

'STOP - CHECK-MARK RELEVANT RECORDS - ANSWER QUESTIONS 
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Sample Search Experiment''- Data Collector-s Scoring Sheet 



o 
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SAMPLE SEARCH EXPERIMENT 

ODD Respondent # . 

W ' ^ Date*_ 



. .' DATA COLLE.aOR'S 'scoring SHEIT 

A. Personal Author Search ' .» 

(CITE). Harold I. Kaplan .Time . 

(IL^) Charles Horace Gray • Time, ' . 



"Al Considering only current -items pubUshed. since 1974, of the information 
retrieved/would you sa/ that the Targeit proportion of relevant informatiorr 
was found using (ILS)(CITE)(NOOIFF)? ^ 

A2 In terms of usef friendliness, did you fiadjt easier to' conduct this 
type of search using. (ILS)(CITE)(NO DIFF)? ' . ' 

A3 ■ In felation to what you were looking for, would you say thaf this type 
of 'search was most satisfactory using (MiS)jCITE)(NO DIFF)? 

B. Corporate Author Search . , ' 

(ILS) Harvard Center for Kiedical .Care (for Gomniunity and) Time_ — _^ 
(CITE) Center for Ulcer Education (Researth and), " . Timfe. 



B2 Considering only current items published since 1974, of the ihfo'rmatipn 
reirieved wSlld you say that the largest proportion of relevant information 
was found using (ILS)(CITE)(NO DIFF)? 

B2 In- tertns'of user friendliness, did you find it easier to conduct this 
ty^e of search using (ILS)(CITE)CNO DIFF)? ' , V 

$3. In relation -to what you were lookingVor, wOuld y^^^ that this type 
pf search was most satisfactory using fILS)(CITE)(NO DIFF)? ; 

. - . • \ ■ ! ~ j ■ - ^'^ , 

C. Title Search . - ' . / . ^ . 

(CITE) Host-Virus Interaction (Molecular Ba'sis 6f) Time, 



(ILS) * Activity in Proteins .(Molecular InteraGtions .and) Time,^ • 

CI Considering only current items publisRid ^ince 1974v of the inf^nnation 
reirieved ^uld you say that the largest ^rofiortion of relevant information 
was found using .(ILS)(CITE)(NO DIFF)? r ^ , ^ .. ' . 

C2" ln°terms of user friendliness, did you find it easier to conduct this 
. . ty^e rff searcMsing (ILS)(CITE,)(NO.DIFF-)? . , . ■ 

03 In relation to what you were loo,king for, would you say that this type 
" - of 'search was most satisfactory using, (ILS) (CITE.) (NO DIFF)? ^ - 
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D. Se tries Search ' 

(ILS) Laboratory and Research l4ethods (in Biology and Medicine) Tiine 

(CITE) Wiley Series in Probability and Mathematical Statistics Time ■ 

(:Applied Probability and Statistics) 

Dl Considering only current items published since 1974, of the information 
retrieved, would you say that the largest proportion of relevant information 
was found using (ILS)(CITE)(NO DIFF)? 

D2 In terms of user friendliness, did you find it easier to conduct this 
type of search using (ILS)(CITE) (NO DIFF)? 

D3 In relation to what .you were looking for, would you say that this type 
of search was most satisfactory using (ILS)(CITE)(NO DIFF)? 

E. Conference Search 

(CITE) Conference on Ambulatory Monitoring Time 

(ILS) Conference- on Recombinant DMA Time 

El Considering only current items published since 1974, of the information 
retrieved, would you say that the largest proportion of relevant information 
was found using (ILS) (CITE) (NO DIFF)? 

■ E2 In terms of user friendliness, did you find it easier to conduct this 
type of search using (ILS)(CITE)(NO DIFF)? 

E3 In relation to what you were looking for, woi/ld you say that this type 
of' search was most satisfactory using (ILS) (CJTE)(NO DIFF)? 

F. Sub.iect Search 

(ILS) Smoking and Lung Neoplasms ^ Time 

(CITE) Cerebrovasc. Disorders and Contra., Oral Time 

(CITE) Toxic Shock Syndrome Time — [ — 

(ILS) Percutaneous Transluminal Angioplasty Time 

Fl. Considering only current items published since 1974, of *he infonnation 
retrieved, would you say that the largest proportion of , relevant information 
was found ysing (ILS)(CITE)(NO DIFF)? 

F-2 In terms of user friendliness, did you find it easier to conduct this 
type of search using (ILS)(CITE)(NO DIFF)? 

F3. In relation to what you were looking for would you say that this type 
of search was most satisfactory using (ILS)(CITE)(NO D^IFF)? 
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ODD ' • 

4. Considering only current items published- since 1974, in general, do 
you think you found more of the information you were looking for using 

(ILS)(CITE)(NO DIFF)? ' 

5. Would you say that system response time was significantly faster 
using (ILS)(CITE)(NO DIFF)? 

6. Would you say that the display of information on the terminal screen 
is most satisfactory using (ILS) (CIT£)(NO DIFF)? 

7. Would you say that the online instructions, prompts and help messages 
are most satisfactory using (ILS) (CITE) (NO DIFF)? 

8. Overall, do you have a preference for (ILS) (CITE) (NO PREF)? 
Would you say your preference is (SLIGHT) (MODERATE) (STRONG)? - 



■ * * .* 

9. What features or attributes of ILS do you find particularly desirable or 
helpful? Of CITE? Please be specific. 



10. What features or attributes of ILS do you find particularly undesirable 
or annoying? Of CITE? Please be specific. 



11. What feat'ures or attributes not now available on ILS would you like to 
•see? On CITE? Please be specific. 
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SAMPLE SEARCH EXPERIMENT 

EVEN Respondent # 



Name. ^^^^ 



' DATA COLLECTOR'S SCORING SHEET 

A. Personal Author Search 

(ILS) Harold I. Kaplan Time _ 

(CITE) Charles Horace Gray ' Time 



Al Considering only current items published since 1974, of the "format on 
retrieved, would you say that the largest proportion of relevant information 
ilrfornd^sing (ILS)(CITE)(NODIFF)? 

A2 'In terms of user friendliness, did you find it easier to conduct this 
type of search using (ILS)(CITE)(NO DIFF)? _ 

A3, in relation to what you were looking for would you say that this type 
of search was most satisfactory using (ILS) (CITE) (NO DIFFJ? 

B. Corporate Author Search 

(CITE) Harvard Center for Medical Care (for Community and) Time_ — 
(ILS) Center for Ulcer Education Research and) Time 



w Considering only current items published since 1974, of the information 
reirifvS wolT^ySu^ay that the largest proportion of relevant information 
was found using (ILS) (CITE) (NO DIFF)? 

B2 'in terns of user friendliness, did you find it easier to conduct this • 
ty^e of search using (ILS)(CITE) (NO DIFF)? 

B3. in relation to what you were looking fo[. ^gjp^p^^ 
of search was rtrast satisfactory using (ILS)(CITE)(NO DIFPJ/ 

C. Title Search 

(ILS) -Host-Virus Interaction (Molecular Basis of) Time 

(CITE) Activity in Proteins (Molecular Interactions and) Time 



^.^X^ ^^^^^^^ 

wirfo^n-d'using (ILS)(CITE)(NO DIFF)? 

C2 In terms of user friendliness, did you find it easier to conduct this 
lli>e of search using. (ILS)(CIT,E)(NO DIFF)? 

Ca. In relation to what you were l^ol^i^pjor, would ^ that this type 
of search was most satisfactory using (ILS)(CITE)(NO DIFF)? 



EVEN 

D, Series Search 

(CITE) Laboratory and Research Methods (in Biology and Medicine) Time_^ 

(ILS) Wiley Series in Probability and N&thematical Statistics Time 

(t Applied Probability and Statistics) 

Dl. Considering only current items published since 1974, of the information 
retrieved, would you say that the largest proportion of relevant information 
was found using (ILS)(CITE)(NO DIFF)? 

D2. In terms of user friendliness, did -you find it easier to conduct this 
type of search using (ILS) (CITE) (NO DIFF)? 

D3. In relation to what you were looking for, would. you say that this type 
of search was most satisfactory using (ILS) (CITE) (NO DIFF)? 

E. Conference Search 

(ILS) Conference on Ambulatory Monitoring Time_ 

(CITE) Conference on Recombinant DNA Time 



El. Considering only current items published since 1974, of the information 
retrieved, would you say that the largest proportion of relevant information 
was found using (ILS) (CITE)(NO. DIFF)? 

^2 In terms of user friendliness, did you find it easier to conduct this 
type of search using (ILS) (CITE)(NO DIFF)? 

E3 In relation to what you were looking for, would you say that this type 
of search was most satisfactory using (ILS)(CITE)(NO DIFF)? 

F. Sub.iect Search 

(CITE) Smoking and Lung Neoplasms Time ° , 

(ILS) Cerebrovasc. Disorders and Contra., Oral Time 

(ILS) . Toxic Shock Syndrome Time_ 

(CITE) Percutaneous Transluminal Angioplasty Time 

Fl Considering only current items published since 1974, of the iT)formation 
retrieved, would you say that the largest proportion of relevant information 
was found using (ILS)(CITE)(NO DIFF)? 

F2 In terms of user friendliness, did you find it easier to conduct this 
type of search using (ILS)(CITE)(NO DIFF)? 

F3. In relation to what you were looking for. would you say that this type 
of search was most satisfactory using (ILS)(CITE)(NO DIFF)? 
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EVEN 

4^ Ansidering only current items published since 1974, in general, do 
you think you found more of the information you were looking for using 
(ILS)(CITE)(NO DIFF)? 

5 Would you say that system response time was significantly faster 
using (ILS) (CITE) (NO DIFF)? 

6 Would you say that the display of information on the terminal screen 
is most satisfactory using (ILS) (CITE) (NO DIFF)? 

7. Would you say that the online instructions, prompts and help messages 
are most satisfactory using (ILS) (CITE) (NO DIFF)? 

8. Overall, do you have a preference for (ILS) (CITE) (NO PREF)? 
Would you say your preference is (SLIGHT) (MODERATE) (STRONG)? 



9. What features or attributes of ILS do you find particularly desirable or 
helpful? Of CITE? Please be specific. 



10. What features or attributes of ILS do you find particularly undesirable 
0.- annoying? Of CITE? Please be specific. 



11. What features or attributes not now available on ILS would you like to 
see? On CITE? Please be specific. 
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APPENDIX G. 

Comparison Search Experiment - Data Collector's Scoring Sheet 
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ODD (Start with CITE) Respondent — _ 

EVEN (Start with ILS) ^ ^^^^ 

SCORING SHEET: COMPARISON SEARCH EXPERIMENT 
***Befo re Starting Search: ^ 

1 What catalog inforietion are you trying to find?. Mark all that apply — 
(A specific book)(Books by a specific author) (Specific information such ^as- 
publisher, date, spelling of a name, etc.)(If a book known to be owned by 
the Library is available for use) (Call number) (Books on a topic or subject) 
— obtain detail s: 



***After Patron Completes Search on CITE and ILS: 

If search query is specific (only), go to item 2 ajd skiP .3 
If search query is on a subject (only), skip item 2 and go to 3 
If search query is both specific and on a subject, go to items d,6 

2. Do you think you found the specific i;t°;'^^tion you were lo^^^^^ for? (YES)(NO) 
If "yes", on which system(s) did you find it? (ILS)ICITEMoOTh; 

3 Do you think you found more of the information you were looking for using 
(ILS)(CITE)(NO DIFF)? 

4 In relation to what you were looking for, would you say your search was 
most satisfactory using (ILS) (CITE)InO DIFF)? 

5. In general did you find it easier to use (ILS) (CITE)(NO DIFF)? ^ . 

6 The next time you need to conduct a catalog search, will you want to use 
(ILS)(CITE)(NO PREF)? 

7 What is the term that best describes your professional role? ("Jealtjj^''® 
Prac )(Health Prof. EdTJTResearcher, Biomed.)(Researcher, Other (Student 

Heafth Prof.)(Student, 6rad.) (Student. Undergrad.) (Other Prof.)(None of the above) 

Are there any additional comments you care to make concerning you^ use of the 
ILS and CITE computer catalog systems? Please be specific. 



THANK YOU very much for your participation in this experiment. 



